RANDOM PARTITIONS AND THE GAMMA KERNEL 



Alexei Borodin and Grigori Olshanski 

Abstract. We study the asymptotics of certain measures on partitions (the so- 
called z-measures and their relatives) in two different regimes: near the diagonal of 
the corresponding Young diagram and in the intermediate zone between the diagonal 
and the edge of the Young diagram. We prove that in both cases the limit correlation 
functions have determinantal form with a correlation kernel which depends on two 
real parameters. In the first case the correlation kernel is discrete, and it has a simple 
expression in terms of the gamma functions. In the second case the correlation kernel 
is continuous and translationally invariant, and it can be a written as a ratio of two 
suitably scaled hyperbolic sines. 
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0. Introduction 

In recent years there has been a lot of interest in understanding the "random 
matrix type" limit behavior of different measures on partitions as the size of parti- 
tions goes to infinity. The most known result is the Baik-Deift- Johansson theorem 
[BD J] that claims that the limit distribution of the (centered and scaled) first part 
of the random partitions distributed according to the so-called Plancherel measure 
is just the same as that of the largest eigenvalue of random Hermitian matrices 
from the Gaussian Unitary Ensemble. 

The goal of this paper is to study the asymptotic behavior of the so-called z- 
measures and their relatives. The asymptotics of the largest parts of partitions 
distributed according to such measures has a representation theoretic meaning: it 
encodes the spectral decomposition of generalized regular representations of certain 
groups into irreducibles. We have computed this asymptotics in the cases of the 
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infinite symmetric group and the infinite-dimensional unitary group in our previous 
work, see [B02], [B04]. The main result of this paper is a complete description 
of the limit behavior of these measures near the diagonal (smallest Frobenius co- 
ordinates) and in the intermediate zone between the diagonal and the edge of the 
partition (Frobenius coordinates of intermediate growth). 

A more detailed description of the content of the paper follows. 

The z— measures. Let Y„ denote the set of partitions of a natural number n and 
Y = Yo U Yi U Y2 U . . . be the set of all partitions. We identify partitions and Young 
diagrams. We consider a Hilbert space H together with a distinguished orthonormal 
basis {xa} parameterized by A e Y. The basis elements may be identified with 
irreducible characters of symmetric groups of arbitrary degree. Next, we construct 
a family of vectors f z ^ in H, indexed by couples (z,£) G C x (0, 1). Set 



where z £ C, z' £ C, < £ < 1, and ( • , • ) denotes the inner product in H. It 
turns out that (f z ,£, f~^) 0, so that the above expression makes sense. Clearly, 

AgY 

Under suitable restrictions on the parameters z, z',^ (for instance, if z' = z), the 
above expression for M ZjZ <^(A) is nonnegativc for any A, so that M ZtZ i^ is a proba- 
bility measure on Y. We call it a z-measure. This is our main object of study. An 
explicit expression for M ZjZ /^(A) is given in §1. 

(n) 

The measures M Z J,. Given n, restrict M ZjZ i£ to Y„ C Y and normalize it so 
that the total mass of Y„ be equal to 1. Then we obtain a probability measure on 
Y„ which turns out to be independent of £; we denote this measure by M^],. The 

initial z-measure M z ^ z >^ may be written as a mixture of the measures M^, with 
varying n, 



(n) 
z,z' 

n=0 



where the coefficients 



■Kin) = (1 - ZZ ' + + 

n! 

are precisely the weights of the negative binomial distribution on Z + with suitable 
parameters. 

Frobenius coordinates. We need the Frobenius notation for Young diagrams: 

A = (pi, ...,p d \q u .. .,q d ), 

where d is the number of diagonal boxes in A, pi is the number of boxes in the ith 
row to the right of the diagonal, and is the number of boxes in the ith column 
below the diagonal. Note that 

Pi > ■ ■ ■ > Pd > 0, qi > ■ ■ ■ > q d > 0. 

An advantage of the Frobenius notation, as compared to the conventional notation 
A = (Ai, A 2 , . . . ), is its obvious symmetry with respect to transposition of diagrams. 
The p^s and q^s are called the Frobenius coordinates of the Young diagram A. 



RANDOM PARTITIONS AND THE GAMMA KERNEL 



3 



Asymptotic problems for random diagrams. Given a probability measures 
on Young diagrams, one may speak about random Young diagrams. A problem of 
interest is to study the asymptotic behavior of M^™ , -random diagrams as n — > oo. 
In the present paper we are dealing with a different but closely related problem: 
the asymptotics of M ZjZ / ^-random diagrams as £ / 1 (the parameters z, z' remain 
fixed). 

There is a number of different limit regimes of the asymptotics. Here we discuss 
three of them: one for the largest Frobenius coordinates, one for the smallest Frobe- 
nius coordinates, and one for the Frobenius coordinates of intermediate growth. 

It is an interesting question how the asymptotics of M ZiZ i£ is related to that of 

in) 

M z z , in each of these regimes. For the first regime the answer is known: the limiting 
random point processes (i.e., measures on point configurations) are different by the 
multiplication by an independent random scaling factor, see [B02, §5] and [BOl, 
§6]. For the third regime, the computation of [Borl, §4.2-4.3], see also [BOl, §11], 
suggests that the asymptotic behavior of the p-coordinatcs is the same for both 
measures. However, this computation is rather involved, and it would be nice to 
have a simpler argument which would also extend to the joint asymptotics of p- and 
q-coordinates. No claims of this kind have been proved yet regarding the second 
regime, but we believe that the corresponding asymptotics of M ZjZ /^ and M^, is 
also the same in this case. 

Asymptotics of largest Frobenius coordinates [B02]. In the first limit regime, 
we look at the largest Frobenius coordinates p\ > p 2 > ■ ■ ■ and q\ > q 2 > . . . . These 
are random variables depending on £ as a parameter. As £ /" 1, we need to normal- 
ize them, and the suitable normalization consists in multiplying all the coordinates 
by (1 — £). In the limit we obtain a couple of random infinite sequences of decreasing 
real numbers, which may be also interpreted as a random point configuration on the 
punctured line R\ {0}, or as a random point process on R\ {0}. This process was 
studied in our previous paper [B02] . We showed that its correlation functions have 
determinantal form with a kernel, which we called the Whittaker kernel, because it 
is expressed through the classical Whittaker function. 

Limit behavior of smallest Frobenius coordinates. In the second limit regime, 
we examine the smallest Frobenius coordinates pd < Pd-i < ■ ■ ■ and qd < qd-i < 
.... Again, these are random variables depending on £, but now no normalization 
is required. In the limit we obtain a couple of random infinite increasing sequences 
of nonnegative integers, say 

< ai < a 2 < ■ ■ ■ , < h < b 2 < ■ ■ . , 

which can be conveniently interpreted as a random point configuration X on the 
lattice U := Z + \ of half-integers, 

X = (. . . , — b 2 — \, —b\ — |, a\ + |, a 2 + \, ■ ■ ■)■ 

Thus we get a random point process on Z', which describes the limit behavior of 
the random Young diagrams near the diagonal. 

A different but equivalent picture of the same limit regime is obtained as follows. 
Set 

x = (xnz' + )u(z'_\x), 
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where 

J' — S -k _3 _1\ J' — f 1 3 5 \ 

— I. * * * 5 2' 2' 2J' + 12' 2' 2 ' ' ' ' J ' 

That is, viewing the points of X as "particles" and those of T! \ X as "holes", the 
configuration X_ is formed by the particles in Z' + and the holes in Z'_ . 

One of the main results of the present paper is a description of both random 
processes on Z'. We show that the correlation functions for each of these two 
processes are given in terms of a rather simple kernel on Z' x Z', which is expressed 
through the Euler gamma function. We call it the gamma kernel (there are two 
versions of the kernel which correspond to the random configurations X and X_, 
respectively). Similarly to the Whittaker kernel, the gamma kernel depends on the 
parameters z, z' . The version corresponding to the random configuration X_ has the 
form 

.gamma, „ i f\ sin(Trz) sin(Trz') V(x)Q(y) - Q(x)V (y) 

7rsm(7r(z — z')) x — y 

where x, y e Z' and 

V{x) = = , Q(x) = 



^T{z + x+\)T(z> + x+\) ' " ^T{z + x+\)T{z> + x+\) 

The m-particle correlation function (m = 1, 2, . . . ) for the random configuration X_ 
is given by 

Prob{X D (xi, . . . , x rn )} = det [^ gamma (x,, Xj \ z, z% 

l<i,j<m 

Here [x\, . . . , x m ) is an arbitrary collection of distinct points in Z'. 

The correlation functions for the random configuration X have the same deter- 
minantal form, only K gamma (x, y \ z, z 1 ) is replaced by another version of the kernel, 
K eamma (x,y | z,z'), see Theorem 3.2 below. 

Asymptotics of intermediate Frobenius coordinates. In the third limit regime, 
we consider Frobenius coordinates with intermediate growth, that is, the p^s and 
qj 's such that 

< Pi , qj < — j as £ / 1. 

We show that in a suitable scaling limit, the asymptotics of the intermediate Frobe- 
nius coordinates is governed by a kernel onRUl (the union of two copies of the 
real line, one is for p-coordinates and the other one is for q-coordinates) . We call 
this kernel the tail kernel. The tail kernel is translationally invariant and it is a 
relative of the famous sine kernel. 

Notice that the tail kernel can also be obtained, via a suitable scaling limit 
transition, from two opposite directions: from the Whittaker kernel, see [BOl, 
§11], and from the gamma kernel, see §6 below. 

Remarks. The measures M^, on the finite sets Y„ first arose in Kerov-Olshanski- 
Vershik [KOV1], in connection with the problem of harmonic analysis on the in- 
finite symmetric group. The asymptotics of the largest Frobenius coordinates of 
M^, -random Young diagrams, as n — > oo, was studied in a series of our papers, 
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summarized in the survey [BOl]. The result provides a description of the decompo- 
sition of the so-called generalized regular representations of the infinite symmetric 
group on irreducible components. 

The z-measures M ZtZ /£ were introduced in [B02]. They enter a wider class 
of Schur measures introduced soon after by Okounkov [Ok2]. The z-measures 
initially served as a technical tool which allowed us to rederive the main results of 
[Bor2], [Bor3] on the limits of the measures in a simpler way. However, the 

z-measures are also interesting in their own right. 

The idea of mixing the measures M^, and replacing the large n limit by the 
£ / 1 limit is similar to the idea of passing to a grand canonical ensemble in 
statistical mechanics. A parallelism between models of statistical mechanics and 
those of asymptotic combinatorics was emphasized by Vershik [V] . 

Comparison with the Plancherel measure. When the parameters z, z' go to 
infinity, the measure M^ n ], degenerates to the Plancherel measure on Y„. 

Similarly, when z, z' go to infinity and £ goes to +0, in such a way that zz't; tends 
to a limit 9 > 0, the measure M ZiZ t£ on Y degenerates to the poissonized Plancherel 
measure Mg with parameter 9. The latter measure is a mixture of the measures 
Af( n \ where the mixing distribution on the n's is the Poisson distribution (the 
weight of n equals e~ e 9 n /nl). The large n limit of the measures can be 

effectively replaced by the large 9 limit of the measures Mg. Due to nice properties 
of the Poisson distribution, both kinds of limit transition turn out to be strictly 
equivalent in various asymptotic regimes (see Baik-Dcift- Johansson [BD J] , Borodin- 
Okounkov-Olshanski [BOO], Johansson [Jl]). 

An important difference between the Plancherel measures and the z-measures is 
that the random Plancherel diagrams have a limit form, as n — > oo or 9 — ► oo (see 
Vershik-Kerov [VK1], [VK2], Logan-Shepp [LS]), while no such form exists for the 
z-measures. On the other hand, the statement of the asymptotic problem concern- 
ing the smallest Frobenius coordinates is the same for both kinds of measures, and 
the answers are formulated in similar terms: for the Plancherel measure, the role 
of the gamma kernel is played by the discrete sine kernel with parameter (see 
[BOO], especially Remark 1.8). Notice that the latter kernel is the degeneration of 
the gamma kernel as the parameters z,z' go to infinity. 

Comparison with the measures given by the Ewens sampling formula. The 

Ewens sampling formula determines a one-parameter family of probability mea- 
sures on Y„ for each n — 1,2, ... : 

ESF< n) (A) = ^ESF< n) (A), AeY„, 

(t)n 

where £(A) is the number of nonzero rows in A, t > is the parameter, and 

(„) . , the number of permutations in S n with cycle structure A 
1 W = n\ 

There is a wide literature concerning these measures, see, e.g., the encyclopedic 
article Tavare-Ewens [TE]. As shown in Kerov-Olshanski- Vershik [KOV1], [KOV2], 

(n) (n) 

both the measures ESFj and the measures 

M zz> 

are involved in harmonic analysis 
on the infinite symmetric group, but they refer to different "levels", the "group 
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(n) 

level" and the "dual level" , respectively. Namely, the measures ESFJ determine 
certain probability measures on a compactification 6 of the infinite symmetric 

(n) 

group, while the measures M z J, determine the so-called spectral measures on the 
dual object to the infinite symmetric group. The measures on 6 are used to build L 2 
Hilbcrt spaces where the so-called generalized regular representations are realized, 
while the spectral measures govern the decomposition of those representations into 
irreduciblcs. 

(n) 

The large n limits of the measures ESF^ in various regimes were extensively 
studied, see, e.g., the monograph by Arratia, Barbour, and Tavare [ABT]. At the 
first glance, the results look quite different as compared with our results for the 

(n) 

measures M\ z ', or M ZtZ i£. Nevertheless, it seems to us that a detailed comparison 
of both families of measures may be of interest since it could lead to a better 
understanding of the nature of probabilistic models related to partitions. 

The zw measures on signatures. By a signature of length N, where N — 
1,2 ... , we mean an ordered A-tuple of nonincreasing integers 

A= (Ai > ••• > X N ), Xi € Z. 

Let SGN(TV) be the set of all such A's. This is a countable set. There is a one-to-one 
correspondence A <-> x\ between signatures A € SGN(TV) and irreducible characters 
of the compact group U(N) of N x N unitary matrices. The irreducible characters 
Xa are given by the (rational) Schur functions s\(u\, . . . , Un) in the eigenvalues 
of a unitary matrix U e U(N). Let Hm be the Hilbert space of functions on the 
group U(N), constant on conjugacy classes and square integrable with respect to 
the normalized Haar measure. The characters \\ form an orthonormal basis in 
Hjsf. Equivalently, can be realized as the space of symmetric functions on the 
torus T N (the product of N copies of the unit circle TcC), square integrable with 
respect to the measure 

]J \ui-Uj\ 2 du, 

l<i<j<N 

where du is the normalized invariant measure on the torus. 

We define a family {J z , w \n} of vectors in H^, where z, w are complex parameters, 
and we set 

M (\\ - (f*MN,Xx)(.X\,fw,-F\N) \^QnN(N\ 
^z,z',w,w'\n(X) - 77 — \ ) X G bGlN (iVj, 

{Jz,w\Ni J w ' : z '\n) 

where ( • , • ) is the inner product in Hpj and (z', w') is one more couple of complex 
numbers. An explicit expression for M z zl _ w w ^ N (X) is given in §7. Under suitable 
restrictions on the quadruple (z, z' , w, w'), this expression determines, for any N, a 
probability measure on SGN(A), which we call the zw-measure. For instance, the 
zw-measures are well defined if z' — z, w' — w, and $t(z + w) > —\. 

The zw-measures arise in the problem of harmonic analysis on the infinite- 
dimensional unitary group, studied in our previous papers [012], [B04]. 

Large N limits of the zw— measures M z z i w w i\ n . Any signature A £ SGN(A) 
can be viewed as a couple (A+, A~) of Young diagrams subject to the condition 
£(X + ) + £(X~) < N, where ^(A ± ) stands for the number of nonzero rows in A ± , 

A = (Xf , Aj, ...,0,0,..., — A 2 , — A 1 ). 
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A problem of interest for the zw-measures is their limit behavior as N — ► oo 
with the parameters z, z' , w, w' being fixed. That is, we ask about the asymptotic 
distribution of the Frobenius coordinates for the random diagrams A+, A~. Again, 
one can consider (at least) three different limit regimes: the largest, smallest or 
intermediate coordinates, respectively. 

The asymptotics of the largest Frobenius coordinates was studied in [012], [B04]. 
Introducing the scaling factor 1/N for the Frobenius coordinates of A+ and A - , we 
obtain in the limit 4 infinite random sequences which can be assembled in a single 
random point configuration living on the real axis with two punctures. We showed 
that this random point process is governed by a kernel, which we called the contin- 
uous hypergeometric kernel for it is expressed through the Gauss hypergeometric 
function. This result leads to a description of the spectral decomposition of certain 
unitary representations of the infinite-dimensional unitary group. 

In the present paper, we arc dealing with the smallest Frobenius coordinates of 
X ± . That is, we study the limit structure of the boundary of the random shape 
A* near its diagonal. Our result is that the limit correlation functions are again 
given by the gamma kernel. The appropriate parameters are (—z, —z') for A+ 
and (—w,—w') for A - . Thus, although in the "first limit regime", the correlation 
kernels obtained from the z-measures and from the zw-measures are different (the 
continuous hypergeometric kernel is on the next level of complexity as compared 
with the Whittaker kernel), the answer in the "second limit regime" is the same. 

It is worth noting that the computations leading to the gamma kernel in the case 
of the zw-measures M ZiZ i iW<w i\n are more complex than those for the z-measures 
M ZtZ t£. Instead of the Gauss hypergeometric function 2 Fi, which is involved in the 
proof for the z-measures, we need to manipulate with the higher hypergeometric 
series — the series 3F2 at the unit argument. 

As for the "third limit regime" , which concerns intermediate Frobenius coordi- 
nates, the answer is conjecturally given by the same tail kernel as for the z-measures. 
We do not prove this fact rigorously but present an argument in favor of it. 

The z— measures on nonnegative signatures. Here we define the third family 
of measures, which are close relatives of the zw-measures described above. Let 
SGN+(A) be the subset of SGN(A) formed by signatures A with \ N > 0. We 
call them the nonnegative signatures. Equivalently, SGN + (A) consists of Young 
diagrams A with < N. We fix two real parameters a > — 1, b > — 1. Let Hjy 
be the Hilbert space formed by symmetric functions on the TV-dimensional cube 
[—1,1]^, square integrable with respect to the measure 

N 

JJ {xi-xjf ■Y[(i-x l ) a (l + x l ) b ■dx 1 ...dx n , si e [-1,1]. 

l<i<j<N i=l 

In Hn, we consider the orthonormal basis {X\' b } formed by the (suitably normal- 
ized) multivariate Jacobi polynomials. Here the subscript A ranges over SGN + (A). 
On the other hand, we introduce a family {/ z \n} of symmetric functions on the 
cube, depending on a complex number z, and we set 

M ZtZ ,, aMN {\) = (/z|Af ;f K ^ J p N) , A e SGN + (7V), 
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where z' is one more complex parameter. If z, z' satisfy certain restrictions, this 
gives us a probability measure on SGN + (A r ), which we call the z-measure on non- 
negative signatures. An explicit expression is given in §8. 

This construction is again motivated by representation theory. Specifically, for 
a few special values of (a, b) , the (suitably renormalized) multivariate Jacobi poly- 
nomials are the irreducible characters of the symplectic or orthogonal groups, or 
else the spherical functions on the complex Grassmannians. Then the z-measures 
^z,z' ,a,b\N naturally emerge in the problem of harmonic analysis for infinite-dimen- 
sional analogs of these classical groups or for the Grassmannians. 1 

For general (a, b), there is no such direct representation-theoretic interpretation. 
Nevertheless, according to the philosophy of the modern theory of multivariate 
special functions (see, e.g. Heckman's part of the book [HS]), there are good reasons 
to work with general parameters (a, b) as well. 

Large N limit of the z— measures M z z * a w^. As in the case of the zw-measures, 
we focus on the "second limit regime" , which, in the present case, concerns the 
smallest Frobenius coordinates of the random diagrams A £ SGN + (iV). And once 
again, it turns out that the limit point process is determined by the gamma kernel. 
The proof involves rather tedious computations with the hypergeometric series 4F3 
at the unit argument. 

Asymptotics of discrete orthogonal polynomials. The heart of the argument 
in the cases of the zw-measures M ZtZ i tW<w i\ N and the z-measures M z >z i a b \ N is a 
computation of the asymptotics of certain discrete orthogonal polynomials of degree 
N — 1 and N as N — > 00. Those are the Askey-Lesky polynomials (which generalize 
the classical Hahn polynomials) in the first case and the Wilson-Neretin polynomials 
(which generalize the classical Racah polynomials) in the second case. Even though 
the weight function in both cases depends on four independent parameters (except 
for N) , the limits of the Christoffel-Darboux kernels in both cases are the same (the 
gamma kernel) , and the result depends only on two of the four initial parameters. 
This suggests that the gamma kernel (and, hence, the tail kernel which is equal 
to its scaling limit) may play a universal role in asymptotics of general discrete 
orthogonal polynomials. 

Recall that, as it was recently shown by Baik-Kricchcrbaucr McLaughlin Miller 
[BKMM, §3.1.1], the discrete sine kernel is the universal microscopic limit of the 
Christoffel-Darboux kernels associated with generic discrete orthogonal polynomi- 
als, near a point where the macroscopic density function is continuous and takes 
any value strictly between and l. 2 

It looks very plausible to us that the gamma kernel and the tail kernel are 
universal microscopic limits, in two different asymptotic regimes, of the Christoffel- 
Darboux kernels for generic discrete orthogonal polynomials near a point where the 
macroscopic density function is discontinuous, takes value on one side of this 
point, and takes value 1 on the other side of this point. The two special cases 
considered in §7 and §8 below provide some evidence in support of this conjecture. 



lr This problem can be stated by analogy with the construction of [012]. Notice that the z- 
measures M z z i a ^ N with a = b = and real z = 2' appeared for the first time in [Pic]. 

2 This means that the point configurations of the corresponding orthogonal polynomial ensemble 
are neither empty (density 0) nor fully packed (density 1) near this point. 
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I. Definition of the z-measures 

As in Macdonald [Ma] we identify partitions and Young diagrams. By Y„ we 
denote the set of partitions of a natural number n, or equivalently, the set of Young 
diagrams with n boxes. By Y we denote the set of all Young diagrams, that is, 
the disjoint union of the finite sets Y„, where n = 0, 1, 2, . . . (by convention, Y 
consists of a single element, the empty diagram 0). 

Given A e Y, let |A| denote the number of boxes of A (so that A e Y| A |), let £(X) 
be the number of nonzero rows in A, and let A' be the transposed diagram. For 
z e C, let 

/(A) 

(z) x = l[(z-i + l) Xi 

i=l 

where (x)k = x(x + 1) . . . (x + k — 1) = Y{x + k)/T(x) is the Pochhammer symbol. 
Note that 

(product over the boxes of A), which implies at once the symmetry relation 

(z) x = (-l)^(-z)y. 

Given A G Y, A =/= 0, we denote by xa the irreducible character of the symmetric 
group S\\\ , indexed by A. For n = 1, 2, . . . , let H n be the space of complex functions 
on S n , constant on conjugacy classes. We introduce an inner product in H n by the 
formula 

ses n 

The characters xa with A e Y„ form an orthonormal basis in H n , so that we may 
write 

#„=0C X A, n=l,2,.... 

agy„ 

We also agree that H is a one-dimensional vector space with basis element denoted 
as X0, (X0,X0)o = 1- 

Given z G C, define a function /i™^ e H n as follows 

/(")(«) = ^thc number of cycles in S; S G S n , U = 1, 2, . . . 

Proposition 1.1 [KOV2, Lemma 4.1.2]. The expansion of in the basis {x\} 
has the form 

AGY„ 

where 

dim A = XA(e). 

For the reader's convenience we outline the proof. 

Proof. Let A be the graded algebra of symmetric functions in countably many 
variables, say yi, yi, . . . , and let A" denotes the nth homogeneous component of A 
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([Ma, §1-2]). Endow A with the canonical inner product ([Ma, §1-4]). Consider the 
characteristic map ch„ , which is a linear isometry between if „ and A™ transforming 
the characters x\ m to the Schur functions s\ ([Ma, §1.7]). First, we check that 

oo 

ch„(/j™)) = the nth homogeneous component of — Ui)~ z ■ 

i=i 

This reduces the claim of the proposition to the expansion 

i=l AeY 1 

which in turn can be deduced from [Ma, chapter I, (4.3)]. □ 

As is well known, dim A coincides with the number of standard tableaux of shape 
AeY (see [Ma, Example 1.7.3]). A number of different explicit expressions are 
known for this quantity. For instance, for any natural k > £(X), 

_ Ui< t<3 < k (^ - X j •+3-i) 



dim A 

,, = det 

A! l<i,j<k 



rii<i< fe (Ai+fc-i)! 



([Ma, Example 1.7.6]). These formulas do not demonstrate the symmetry dim A = 
dim A'. There are two other formulas which are symmetric: the hook formula 
([Ma, Example 1.5.2]) and the expression in terms of Frobenius coordinates, see the 
beginning of §3 below. 
Let us agree that 

= X e H Q . 

Proposition 1.2. For any z G C and any £ £ (0, 1) ? we have 

Ell/i rl) Hn|7 <+ - 

so that the formal sum 



n=0 



n=0 

is a well-defined element of the Hilbert space 

H := H Q ® H x ® H 2 ® ... 

Proof. We will prove that 

= (zz) n = (zz)(zz + l)...(zz + n- 1). 

Since the series 

n\ ? 

converges for < £ < 1 , the claim of the proposition will readily follow. 



By the definition of /j™' 



ll/i ^IL~ (fzz )X(n))n 



where (n) is the partition (n, 0, 0, ... ) (the corresponding character is simply the 
constant function 1). Then the result follows from Proposition 1.1. □ 
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Proposition 1.3. For any z, z' e C and any £ G (0, 1), we have 



(/*,£, /F, £ ) = (!-0 



— zz' 



where ( • , • ) is the inner product in H . 

Proof. The same argument as in the proof of Proposition 1.2 gives 



{fi n \f^)n = {zz') 



Therefore. 



CO 

(/^/^) = E(/i" ) '/fr ) ) 

n=0 



E 



CO- 



n! 



■??, 



= (i-0 



— zz' 



□ 



Definition 1.4. (i) Let z, z' e C and £ e (0, 1). For any A e Y we set 



M 2 ,^(A) - 




Notice that the denominator is nonzero (Proposition 1.3), so that the whole expres- 
sion makes sense. Since {xa}agy is an orthonormal basis in H, we have 



From Propositions 1.2 and 1.3 we obtain an explicit expression for M z ^^: 



(ii) Under suitable restrictions on the triple (z, z', £) the quantities M Z;2 / > {(A) are 
nonnegative for all A. Then M z>z r^ is a probability measure on the countable set 
Y, which we call the z-measure on Y with parameters z,z',£. The nonnegativity 
property holds, for instance, if z' — z; other sufficient conditions are given in 
Corollary 1.9 below. The definition of the z-measures M ZiZ i^ was given in Borodin- 
Olshanski [B02]; see also [B03]. It is a modification of a construction due to 
Kerov-Olshanski-Vershik [KOV1], [KOV2]. They z-measures enter a larger class 
of Schur measures as defined by Okounkov [Ok2]. 

Example 1.5. Assume z = k, z' = I, where k,l are natural numbers. Then 
{z)\ vanishes unless £(X) < k, and likewise {z')\ vanishes unless ^(A) < I. If 
£(A) < min(fc, I) then both (z)\ and (z')\ are strictly positive. It follows that 
Mk,i,£ is a measure supported by diagrams A with at most min(fc, I) rows. This 
z-measure can be obtained by the following construction. 

Let S(C k ® C l ) be the symmetric algebra of the vector space C k <g> C'. This 
is a graded space. Let be the operator in S(C k C ) taking value £" on the 
nth homogeneous component. On the other hand, as a bi-module over GL(k,C) x 
GL(l,C), the space S(C k ® C l ) is the multiplicity free direct sum of irreducible 
bi-modules of the form V\ t k ® V^j, where A ranges over the set of Young diagrams 



E M *.*'.«( A ) = 1 - 



AeY 
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with ^(A) < min(A:, I) and Vx,k denotes the irreducible polynomial GL(k, C)-module 
indexed by A. Given A with ^(A) < min(fc, I), denote by I\ the projection onto the 
component Vx.k ® Vx.i- Then we have 

M kM (X) = tr [^ A) , £(X) < mm(k, I). 

A closely related interpretation is as follows. Consider the set Mat(fc, Z;Z+) of 
k x I matrices with entries in Z+. The Robinson-Schensted-Knuth algorithm (RSK, 
for short) determines a projection of Mat(fc, I; Z + ) onto the set of Young diagram 
with at most min(fc,Z) rows (see e.g. Sagan [Sa, Theorem 4.8.2]). Let Mk,i,£ be 
the probability measure on Mat(fc, I; Z + ) defined by the condition that the matrix 
entries are independent random variables distributed according to the geometric 
distribution with parameter £. Then the push-forward of Mk,i,£ under RSK is 

Asymptotics of the first part of random partitions distributed according to Mk,i,£, 
has been thoroughly studied by Johansson [Jl]. 

Example 1.6. Once again, let k,l be two natural numbers, and take z = k, 
z' = —I. Then (z)\(z')\ vanishes unless £(X) < k and £(X') < I, that is, A must be 
contained in the rectangular shape of size kxl. If this condition is satisfied then the 
sign of (z)x(z')\ equals (— 1)I A L Assume now that £ < (we temporarily abandon 
the restriction £ € (0,1)). Then the factor £' A ' in Definition 1.4 will compensate 
the oscillation of the sign of (z)\(z')\, and we again obtain a probability measure, 
Mk,-i£. Note that it is supported by a finite set of Young diagrams. 

Both interpretations of the measure Mkj t £ given in Example 1.5 can be extended 
to the measure Mk t -i£, with suitable modifications. Namely, the symmetric algebra 
S(C k <g>C l ) is replaced by the exterior algebra A(C fc ® C l ). Let be the operator 
in this graded space taking value (— £)" on the nth homogeneous component. The 
exterior algebra decomposes into irreducible bi-modules of the form V\,k <8> Vx>,i- 
Let I\ denote the projection onto V\.k <8> Vyj. Then 

Mk,-uW = ^. , £(X)<k, £(\')<l. 

Further, consider the (finite) set Mat(fc, I; 0/1) of k x I matrices with entries in 
{0, 1}. We equip Mat(fc, I; 0/1) with the probability measure Mk,-i,£ such that the 
matrix entries are independent and identically distributed according to 

P"*(o) = ^ Prob (i) = lf^. 

Instead of the Robinson-Schensted-Knuth algorithm we apply its dual version 
(dRSK), see [Sa, Theorem 4.8.5]. Taking the push-forward of Mk,-i,£ with respect 
to dRSK we obtain Mk-i£. 

Asymptotics of the first part of random partitions distributed according to 
Mk,-i,(, has been thoroughly studied by Gravner-Tracy-Widom [GTW]. 
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Example 1.7. Let the parameters z, z', £ vary in such a way that 

\z\^oo, \z'\^>oo, C^O, zz'^6>0. 
Then we obtain in the limit the poissonized Plancherel measure with parameter 0, 

Asymptotics of the Plancherel measure has been studied by many authors, see e.g. 
[LS], [VK1], [VK2], [BDJ], [BOO], [J2], [Ok3] and references therein. 

Proposition 1.8. Let z,z' be nonzero complex numbers. The quantity (z)\(z')\ 
is nonnegative for any A G Y if and only if one of the following three conditions 
holds: 

(i) The numbers z, z' are not real and are conjugate to each other. 

(ii) Both z, z' are real and are contained in the same open interval of the form 
(to, to + 1), where to e Z. 

(iii) One of the numbers z, z' (say, z) is a nonzero integer while z' has the same 
sign and, moreover, \z'\ > \z\ — 1. 

Proof. Consider two cases: (1) both z, z' are not integers; (2) at least one of z, z' 
is an integer. 

(1) In this case, the quantity (z)\(z')\ does not vanish. It is strictly positive for 
all A e Y if and only if (z + k)(z' + k) > for any integer k, which is equivalent to 
(z, z') satisfying (i) or (ii). 

(2) Without loss of generality we may assume that either z is an integer and z' 
is not, or both z,z' are integers and \z\ < \z'\. Next, by virtue of the symmetry 
(z)\(z')\ = (—z)\i(—z')\i, we may assume z = m = 1,2, . . . (note that z = is 
excluded by the hypothesis). Then (z)\ vanishes if £(X) > to, and is strictly positive 
if ^(A) < to. Therefore, the quantity (z)\(z')\ is nonnegative for all A if and only 
if ( z ')\ > for all A with 1(A) < to, which means that z' must be a real number 
> to — 1. (Note that z' ^ m — 1 because of the assumption \z\ < \z'\.) □ 

Corollary 1.9. Let z,z' satisfy one of the conditions (i), (ii), (iii) of Proposition 
1.8, and let < £ < 1. Then the z-measure M ZjZ /^ with parameters z,z',£ is well 
defined as a probability measure. 

Notice the symmetry relation 

M z , z ,, t (\) = M_ 2 ,_^(A'). 
Henceforth we assume that the parameter £ belongs to the open interval (0, 1). 

2. The hypergeometric and gamma kernels (first form) 

Let Z' denote the lattice of proper half-integers: 

Z' = Z+i = {±I,±§,±§,...}. 

Consider the space of all subsets IcZ'. By assigning to any IcZ' its charac- 
teristic function we identify that space with the space {0, 1} Z of all doubly infinite 
binary sequences indexed by elements of the lattice Z': 

(...,o_ 3/ 2,a_i / 2 I ai /2 ,a 3 /2,...)> a x e{0,l} Vx e U. 
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We endow the space {0, 1} Z with the product topology, which makes it a compact 
topological space. 

To any diagram A € Y we assign a subset X(A) C U , 

X(\) = {K-i + ± | i = 1,2,...}, 

which wc identify with the corresponding binary sequence (a X (X)) X £Z> (so that 
a x = 1 if x equals Xi — i + \ for some i, and a x = otherwise). Thus, we obtain an 
embedding Y {0, 1} Z . For instance, the empty diagram turns into the binary 
sequence (. . . Ill | 000 . . . ), and the diagram A = (3, 1) G Y 3 turns into the binary 
sequence (...11101 | 00100...). The binary sequence (a x (X)) xe z' has a simple 
geometric meaning: given k = 1,2,..., the digit a±(/c-i/2) is 1 or depending on 
whether the kth segment of the boundary of A above/below the diagonal is vertical 
or horizontal. 

Note that image of Y is dense in {0, 1} Z , so that {0, 1} Z is a compactification 
of the discrete space Y. 

Definition 2.1. Let P be an arbitrary probability measure on the compact space 
{0, 1} Z . The m-point correlation function (m = 1, 2, . . . ) of P, denoted as p m ( ■ \ 
P), is defined on m-point subsets X — {xi, . . . , x m } C U . The value p m {x\, ■ ■ ■ , x m \ 
P) at X is the probability that the random (with respect to P) set contains X. 
Equivalently, this is the probability that the random (with respect to P) binary 
sequence has l's at the positions x\,...,x m - □ 

Notice that P is uniquely determined by its correlation functions. Indeed, using 
the inclusion/exclusion principle we can compute the P-measure of any cylinder 
set of the form {(a x ) \ a yi =£!,..., a Vm — e m } with arbitrary j/i, . . . , y m £ U and 

£\ , . . . , £ m 0,1. 

It turns out that the correlation functions of the z-measures can be explicitly 
computed. 

Theorem 2.2. Assume that both z, z' are not integers. That is, (z, z') is subject to 
one of the conditions (i), (ii) of Proposition 1.8, but not to the condition (iii). Let 
P_ z z , ^ be the push-forward of the z-measure M z , z i^ under the embedding A X(\) 

of the discrete space Y into the compact space {0, 1} Z . 

The correlation functions of P_ z z , ^, as defined in Definition 2.1, have determi- 
nantal form 

p m (x 1 ,...,x m | P z , A = det [K(xi,Xj \z,z',£)], 

±<i,j<m 

m=l,2, xi,...,x m eZ', 
where K_{x,y \ z,z',£) is a function on U x Tl not depending on m. Specifically, 

K { x,y\z,z',Q= P ^- Q ^\ 

x-y 

with 

p{x) = p(x i z, z\ o = (zz') 1/4 r /2 (i - o {z+z ' )/2 

T(z + x + i)r(z' + x + \) \ 1/2 F(-z, -z'; x+^^j) 



T(z + l)T(z' + l) J r(a; + i) 
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Q(x) = Q(x I z, z', = (zz') 3/4 C (a;+1)/2 (l - {z+z ' )/2 ~ 1 

T(z + x + \)Y{z' + x + \)\ 1/2 F(-z + 1, -z' + hx + §; A) 



r(z + i)r(z' + i) y r(ar + §) 

where F(a, b; c; to) stands for the Gauss hyper geometric function. 

Comments. 1. The ratio F(a, b; c; w)/T(c) is an entire function in the parameter c, 
see Erdelyi [Erl, 2.1.6]. Next, under our assumptions on the parameters z, z' , 



T{z + x+l)T{z' + x+\) 



> 0. 



r(z + i)r(z' + i) 

This implies that P(x) and Q(x) arc well defined on the whole lattice U . 

2. Moreover, the expressions of the functions P(x), Q(x) are also well defined in 
a neighborhood of Z' in C, and these are analytic functions. This makes it possible 
to define the value of ratio (P(x)Q(y) — Q(x)P(y))/(x — y) on the diagonal x = y, 
by making use of the l'Hospital rule. 

3. Notice that 

Q(x\z,z',o = ( (jTTx^TTj j p ( x + 1 \ z -^ z '-^0- 

4. We call K_(x,y \ z,z',£) the discrete hypergeometric kernel. 

Proof of Theorem 2.2. As is shown below (Corollary 4.3), Theorem 2.2 is equivalent 
to Theorem 3.2, and the latter theorem was proved in Borodin-Olshanski [B02]. 
On the other hand, Theorem 2.2 can be proved directly, see Okounkov [Okl] and 
Borodin-Okounkov [BOk, Example 3]. □ 

Recall that the parameter £ of the z-measure ranges over the open interval (0, 1). 
What happens when £ tends to one of the endpoints 0, 1? From the definition of 
the z-measures it easily follows that as £ tends to 0, the z-measure tends to the 
Dirac measure at 0, while the limit as £ tends to 1 is the zero measure: 

limM Zi ^(A) = 0, VAeY. 

However, the £ /* 1 limit becomes nontrivial when instead of Y we take its com- 
pactification {0, 1} Z ' D Y. 

Theorem 2.3. Let (z, z 1 ) and P z , ? be as in Theorem 2.2. As^ / 1, the measures 
E-z,z',i wea kly converge to a probability measure £f™ a on {0, 1} Z . The correlation 
functions of the limit measure have determinantal form, 

p m ( Xl ,...,x m | P s ~ ) = dot [X s ™ (x u x 3 | z, *')], 

l<i,j<m 

m=l,2, xi,...,x m eZ', 

where x gamma (x , y \ z,z',£) is a function on Tl x Z' not depending on m. Specifi- 
cally, 

K~(x,y\z,z') = S ™ { ™ ) ™ {7r *'l 
7rsm(7r(z — z'jj 

x {r(z + x + \)V{z' + x + \)T{z + y + \)Y{z' + y + ^y 1/2 

T(z + x + \)V{z' + y+\)- T(z' + x + \)Y{z + y + |) 
x-y 
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Comments. 1. The expression in the curved brackets is strictly positive because of 
our assumptions on the parameters z,z'. 

2. If z = z', which is only possible when z = z' € R\Z, then the above expression 
takes a simpler form 



K psi (x,y\z) 



sin(7T2;) \ 2 ip(z + x + \) - ip(z + y + \) 



x-y 



where ip(u) = T'(u)/T(u) is the logarithmic derivative of the T-function. 
3. On the diagonal x — y we have 

k~(x, x i z, Z >) = ;;g ( f_ (7r ;jj + * + d - w + * + 



K psi (x,y\z) 



x=y 



( sin(7T2:)\ 2 , , , 



4. Wc call K_ eamma (x, y \ z, z') and K_ psi (x, y | z) the gamma kernel and the psi 
kernel, respectively. 

Proof of Theorem 2.3. We will show that the discrete hypergeometric kernel K_ (x , y \ 
z,z',£) of Theorem 2.2 has a pointwise limit as £ f 1, and the result is the 
gamma kernel. This will imply Theorem 2.3. (Notice, however, that the functions 
P(x | z, z', £) and Q(x \ z, z', £), in general, do not have limits as £ / 1.) 
We use the formula (see Erdelyi [Erl, 2.1.4 (17)]) 

W) F(a ' b; C; W) = ^rl^ ^ 1 - c + a; 1 - 6 + a; 
+ r %a)r(c-l) b F{bl 1 " c + b] 1 ~ a + b] w e c ^ [0 ' 

For fixed a, b, c and large negative w, we write 

F(a, 1-c+a; 1-b+a; uT 1 ) = l+O^" 1 ), F(b, 1-c+b; 1-a+b; uT 1 ) = l+0(w _1 ), 
which gives 



r( c ) ^'^>~; r(6)r( c -o) v v " r(o)r( c -&) 

(2.1) 

Specializing this simple estimate to 

a = — z, b = —z', c = x + \, w 



2 £-1 
and to 

3 £ 
a = — z + 1, b = —z' + l, c = x+—, w 



2 £ - 1 
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we obtain (below we denote by S\ , 62 , ■ ■ ■ suitable quantities of the type 1 + O ( 1 — £) 
whose precise form is unessential) 

P{x I z, z , £) = [zz ) 1 



r(z + i)r(z' + i) 

f r(z- zQCi-Q^'-*)/ 2 r(z / -z)(i-Q( z ~ z ')/ 2 I 
x \ r(-z')r(z + a; + i) 1+ r(-z)r(z' + x + i) 2 J 

[ r(z-z')(l-fl(''-»>/ 2 IV - z)(l - 1 

\r(-z' + i)r(z + ?y + i) 3 + r(-z + i)r(z' + y + i) 4 J 

Substituting these expressions into P{x)Q{y) one sees that the term involving the 
factor (1 — £) z ~ z or (1— S.Y ~ z wm cancel with the corresponding term in Q(x)P(y), 
within a quantity of the form (1 — ^) ± ^ _z -*0(1 — £). Such a quantity is negligible, 
because \$t(z — z')\ < 1, as it follows from our assumptions on z,z'. Thus, only 
terms not involving the factors (1 — ^) ±( ^ z ~ z ) survive in P(x)Q(y) — Q(x)P(y). 
Writing these terms down we get 

P(x | z, z', Q(y | z, z', - Q(x | z, z', P(y | z, z', 
sin(7rz) sin(7rz') r , , , , . 10 -i/2 

= 7rsin(7r(z-^)) { ( * + ^ + ^ ( + * + ^ " + V + ^ +V+ ^ 

x {r(z + x + i)r(z' + y + 1) - r(z' + x + ±)r(z + j, + ±) + (i)} . 

This proves the claim of the theorem for x ^ y and z ^ z'. To remove the restriction 
x 7^ y we remark that the above formula holds not only for x,y on the lattice U 
but also in a suitable neighborhood U of the lattice U in C. Moreover, one can 
prove that the remaining term o(l) admits a uniform bound provided that x, y 
range over compact subsets in U x U. Thus, as £ — > 1, the left-hand side (which 
is a holomorphic function in U x £/, vanishing on the diagonal x = y) converges 
to the right-hand side with the remaining term (which has the same vanishing 
property) removed, uniformly on compact sets. This makes it possible to remove 
the indeterminacy on the diagonal x = y using the L'Hospital rule. 

Finally, to handle the case z — z' we apply a similar argument of analytical con- 
tinuation, using the fact that the expressions for the kernels are locally holomorphic 
functions in (z, z') □. 

3. The hypergeometric and gamma kernels (second form) 

Recall the definition of the Frobenius coordinates of a nonempty diagram A € Y: 
these are the integers p\ > ■ ■ ■ > pd > 0, q\ > ■ ■ ■ > qd > 0, where d is the number 
of boxes on the main diagonal of A and 

Pi = Xi-i, q i = X' i -i, i = l,...,d. 

Any collection of integers p\ > ■ ■ ■ > pd > 0, q\ > ■ ■ ■ > qd > corresponds to a 
Young diagram. The transposition A i— » A' corresponds to interchanging pi <-» ^ . 
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In terms of Frobcnius coordinates, the expression for the z-measure, see Definition 
1.4, can be rewritten as follows 

M Z , Z ^(A) = (1-£) ZZ '£ |A| {zz') d 

d 2 

x f[(z + l) Pi (z' + l) Pi (-z + l) qi (-z' + l) qt ■ , 

where 

dim A _ Ui<i<j<d(Pi ~ Pj)(li ~ Qj) 
|A|! ni <i,j<d 

{ Pi + q 3 + 1) 111 
To any diagram A € Y we assign a finite subset X(X) C Z': 

I(A) = I + (A)UI_(A), 
X+(X) = {p ly . . . ,p d } c Z' + , X_(X) = {-§!, ...,-&} C Z'_, 

where 

Pi = Pi + 5, 5i = ft + |, i = l, ...,d, 
are the modified Frobcnius coordinates of A and 

7' — { _ 5 _ 3 _ 1 T. 7/ _ r 1 3 5 \ 

— l. * * * 5 2 ' 2' 2J' + ~ I 2 ' 2 ' 2 ' ' ' ' J ' 

By convention, X(0) = 0. Note that A is uniquely determined by X(X), so that 
the correspondence A X(X) is an embedding of Y into the space {0, 1} Z . 

Proposition 3.1. The correspondence X i— > -^(A), defined above, and the corre- 
spondence X i ► X(X), which was defined at the beginning of $2, are related to each 
other as follows. For any A G Y, 

X(X) = X(X) A Z'_, X(X) = X{X) A Z'_, 

where the symbol A denotes the symmetric difference of two sets. 

Proof. This can be proved using a simple geometric argument, cf. Borodin Olshanski 
[B04, §4]. Notice that the claim is equivalent to the classical Frobenius lemma, see 
Macdonald [Ma, Example 1.1.15 (a)]. 3 □ 

In terms of binary sequences, the claim of Proposition 3.1 can be restated as 
follows. Let a i ► cl° denote the involutive homeomorphism of the space {0, 1} Z 
which applies the transposition <-> 1 to all digits indexed by negative semi- 
integers. Then we have X(X) = (X{X))°, 2L(X) = (X(X))°. 

Let P z ,z',£ be the push-forward of the measure M ZiZ >£ under the embedding 
Y <^-> {0, 1} Z defined by the correspondence A i— ► X(X). Then, by Proposition 3.1, 
Pz,z',i coincides with image of the measure P_ zz ,^ under the involution a i— > a°. 
We aim to write down the correlation functions of P z ,z',£- 



3 This claim was exploited in Borodin-Okounkov-Olshanski [BOO, (1.2)]. In that paper, X(X) 
and X(\) were denoted as V(X) and Fr(A), respectively. 
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Theorem 3.2. Let (z,z') be as in Theorem 2.2. The correlation functions of the 
measure P z ,z>,£ have determinantal form 

p m {x u ...,x m | P x , x ;t) = det [K(xi,Xj)\, m=l,2, 

l<« ;< ;<m 

where the kernel 

K{x,y) = K(x,y \ z,z',£) 

on U x U is defined by the following formulas depending on the sign of x and y. 

• For x > 0, y > 0: 

Pjx | z, z', Q Q(y | z, z', Q - Q(x I z, z', Q P(y | z, z', fl 

x- y 

• For x > 0, y > 0: 

P(x | z, z', Q P(-y | -z, -z', Q + Qfo | z, z', Q Q(-y | -z, -z', fl 

x- y 

• For x < 0, y > 0: 

x - y 

• For x < 0, y > 0: 

^(-s I -z', Q Q(-y I -z, Q - Q(-a: I ^ fl P(-y I -z, -z', Q 

-x + y 

/fere P and Q are ifte functions introduced in Theorem 2.2. 

Comments. 1. Notice that i4T(y, x) = sgn(x) sgn(y)F'(x, y). 

2. The indeterminacy 0/0 on the diagonal x = y is removed by making use of 
the l'Hospital rule. 

Proof. Sec Borodin-Olshanski [B02, Theorem 3.3]. □ 

Actually, Theorem 3.3 in [B02] contains a stronger claim (see Theorem 3.4 
below). In order to state it, we introduce a kernel A on U + x Z'_ by 

^-^/ysin(7rz)sin(7rz') 
A(x, y | z, z , = * 



^(z + x+i^z' + x + i) ^/r(-z-y+i)r(-z'-y+I) x 



T(x + i) T(-y+i) x-y 



xeZ' + , ye 
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Proposition 3.3. The kernel A(x,y | z,z',£) is of trace class, i.e., the correspond- 
ing operator i 2 (U_) — > £ 2 (Z' + ) is of trace class. 

Proof. First of all, note that the denominator x — y does not vanish, because x~y > 
1. Observe that 



\A(x,y | z,z',0\ < oo. 



Indeed, in the expression for the kernel, the ratios of gamma factors have at most 
polynomial growth, 



yjv{z + x + \)Y{z' + x+^) 



T{x+\) 



T{-z-y+\)Y{-z'-y+\) 



\y\ 



-(z+z')/2 



+00 



y -> -oo 



(3.1) 



while ^ x -y)/ 2 = ^( x +l2/l)/2 has an exponential decay. 

Now the claim follows from a well-known sufficient condition: an infinite matrix 
A = [Aij] is of trace class if the sum l^ul i s finite. Here is a simple argument 
that justifies the sufficiency. 

It is enough to show that ||-<4||i < where ||^4||i is the trace norm. We 

have 

p||i=sup|tr(AB)|, 

B 

where B ranges over the set of all (say, finite-dimensional) matrices with ||B|| < 1, 
and | |.B 1 1 is the ordinary norm. But 

|tr(AB)| = ^ El^l ' \ B *\ < 

because ||B|| < 1 implies \Bji\ < 1. □ 
Next, introduce a kernel L on U x U by 

f 0, x > 0, y > 



L(x,y | z,z',£) = < 



A(x,y | z,z',0, x > 0, y < 
-A(y,x | z,z',£), x < 0, y > 
0, x < 0, y < 0. 



(3.2) 



Let L be the operator in the Hilbert space £ 2 (Z') defined by this kernel. By Propo- 
sition 3.3, L is of trace class, so that det(l + L) makes sense. 
It is readily checked ([B02, Proposition 3.1]) that 



Af ZiZ / i£ (A) = 



x,yeX(\) 



det(l + L) 

By a general claim (see [B02, §2]), this implies that 



Pm(xi 



, . . . , 



p z,z',d = det 



l + L 



A G Y. 



(Xi , X j ) 
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Theorem 3.4. Let L be the operator in £ 2 (Z') with kernel L(x,y | z,z',£). The 

kernel K(x,y | z, z',£) is precisely the matrix of the operator -. 

1 + L 

Proof. Sec Borodin-Olshanski [B02, Theorem 3.3]. □ 

The next claim is a counterpart of Theorem 2.3. 

Theorem 3.5. As £ / 1, the measures P z ,z',£ weakly converge to a probability 
measure ps™ ma on {(^ 1} Z . The correlation functions of the limit measure have 
determinantal form, 

Pm (x u ...,x m \ Pl~) = det {Kt—(x t , x 3 | z, z% 

l<z,j<m 

m = i,2, n,...,i m eZ', 

where the kernel -ftT gamma (x, y \ z,z') on U x 11 , which is equal to the pointwise 
limit of the kernel K(x,y \ z,z',£) as £ / 1, is given by the following formulas 
depending on the signs of the arguments x,y. 

• For x > 0, y > 0, the kernel is given by same expression as in Theorem 2.3: 

sin(7rz) sin(7rz') , , . , , . 1n1 -i/2 

vL^z-z')) {T(Z + X+ ^ )r(Z + X + ^ )r(Z + V + ^ )r(Z + V+ ^ 
T(z + x + \)T{z< + y+\)- T(z' + x + l)r(z + y+j) 
x-y 

• For x > 0, y < 0: 

( r( = + * + = )r ^' + 1 + ^ )r <-= - » + -y + i>y 1 ' 2 

sin(7rz)r(z + x + \)T{-z - y + |) - sin(7^z / )^(z , + x + §)r(-z' - y+ |) 

a; - y 

• For x < 0, y > 0: 



A/sin(7rz) sin(7rz') , , , ._ , , , ._ , , 1n -,-i/2 

7rsin(7r(z- z0) { ( ~" " * + ^ ( ~ * + ^ { * + V + * W + y+ ^ 
sin(7rz)r(-z - x + \)T(z + y + |) - sin( 7 rz / )r(-z / -x + l)r(z' + y + |) 

• For x < 0, y < 0: 

sin(7rz) sin(7rz') , , . , , , 1n1 -i/2 

^sin^z-z')) { " * + ^ ( " ~ ^ + * } ( ~* ~ ^ + ^ ( " ^ + ^ 
x T(-z -x+ \)T{-z> -y+\)- T{-z> -x+ |)T(-z y + |) 

x - y 

Proof. The case x,y > was proved in Theorem 2.3. This immediately implies 
the case x, y < 0, because of an obvious symmetry of the formulas of Theorem 
3.2 (changing the signs of x,y is equivalent to changing the signs of z,z'). In the 
remaining two cases we argue just as in the proof of Theorem 2.3. □ 
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4. The relation between two forms of kernels 

Our next goal is to describe a relation between the two types of the discrete 
hypergeometric kernel K_(x, y \ z, z', £) and K(x, y \ z, z', £) and, similarly, between 
the two types of the gamma kernel X gamma (x, y \ z, z') and K^ mma (x, y j z, z 1 ). 

Given an arbitrary kernel K(x,y) on U x Z', we assign to it another kernel, 

K°(x y) = { K( - X ' y ^ X> °> 
K(x,y), x<0, 

where S xy is the Kronecker symbol. Slightly more generally, given an arbitrary map 
e : U -> K*, we set 

K°^(x,y)=e(x)K°(x,y)e(y)- 1 . 

Proposition 4.1. Let P be a probability measure on {0, 1} Z and P° be its image 
under the involutive homeomorphism a t-^ a° of the space {0, 1} Z , introduced after 
Proposition 3.1. Assume that the correlation functions of P have determinantal 
form with a certain kernel K{x,y), 

Pm (x u ...,x m | P) = det [K{xi,Xj)\, to = 1,2,.... 

l<i,j<m 

Then the correlation functions of the measure P° also have a similar determinantal 
form, with the kernel K°(x,y) as defined above or, equally well, with the kernel 
K°- £ (x,y), where the map e : U — > R* may be chosen arbitrarily, 

p m (xi, ■ ■ ■ ,x m | P°) = det [K°(xi,Xj)]= det [K°' e (xi, xj)}, 

TO = 1,2,... . 



Proof. The factor e( • ) does not affect the values of determinants in right-hand side 
of the above formula, so that we may take e( • ) = 1. Then the result is obtained 
by applying the inclusion/exclusion principle, see Proposition A. 8 in Borodin- 
Okounkov-Olshanski [BOO]. □ 

Theorem 4.2. The kernels K_(x,y | z,z',£) and K(x,y | z,z',£), introduced in 
$2 and §5, respectively, are related to each other by the transformation K K°' £ , 
where 

e(x) = { h ^ Z +' 

I (-l) fe , ar = -(fc+|)eZ'_, k = 0,1,2,.... 

Comments. 1. Since the kernels in question are associated with the measures P_ z z , ^ 
and P z ,z>,£, which are related to each other by the involution, the claim of the 
proposition is not surprising, in view of Proposition 4.1. The point is the explicit 
form of the factor e( ■ ). 

2. The claim of the theorem generalizes Lemma 2.5 in Borodin-Okounkov- 
Olshanski [BOO]. 

Before giving a proof let us state a corollary. 
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Corollary 4.3. Theorem 2.3 and Theorem 3.2 are equivalent. 

Proof. Indeed, this follows from Proposition 4.1 and Theorem 4.2. □ 

Proof of Theorem 4-2. (a) Let us check the desired relation between K{x, y \ z, z', £) 
and K(x,y \ z,z',£) for an arbitrary couple x,y outside the diagonal x = y £ U. 
The classical transformation formula [Erl, 2.8(19)] implies 



= (-l) k+i e +1 (l-i) a+b - 1 (a) k+l (b) k+l 

c— — k 

1 _ . . . . 



(4.1) 



"rtH2j f(1 - fl ' 1 - 6;H2 ^ ) 

for any a, b G C and any k = — 1, 0, 1, 2, 

Prom this we derive 

P(x | z, z', = (-lf-VSQ^a; | _ Z) £), a; e z'_ 
Q(jc | z,z',£) = (-l) 2+1/2 P(-a; | -z,-z',£), & € Z'_. 

This readily implies the relation in question. 

(b) Consider now the case x = y G Z'. We have to prove that 

K{x,x | z,z',£) = 1-K(x,x | z,z',£), x e Z'_. 
First, we will prove that 

^ ar|«,«',0 = -^ a: |«, 0, * G Z'_ . 
By virtue of Proposition 4.5 below, this is equivalent to 

P(.t | z, z', f )Q(a; | z, z', = -P(-a; | -z, -z', f I -z', £), a; e Z'_ 

which in turn follows from formulas (4.1) above. 

(c) To conclude the proof it suffices to prove that 

lim K(x, x | z, z', £) = lim (1 — K_(x, x | z, z', £)), a; G Z'_ . 

By virtue of Theorem 2.3, this means 

X gamma (a;,a; | z, z') + ^ gamma (-a;, -x | -z, -z') = 1, x G Z'_ . 

Using Comment 3 to Theorem 2.3 we reduce this to 

sin(7rz) sin(7rz') , , . , 

7rsm(7r(z - z')) 1 z z 

- -0(-z -x + \) + ip(-z' - x + §)} = 1, 

which is verified using a well-known relation for the ^-function [Erl, 1.7.1(8)]: 

"4>(a) — ip(l — a) = — 7rctg(7ra). □ 



The counterpart of Theorem 4.2 for the gamma kernels is 
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Theorem 4.4. The kernels K gamma (x,y | z,z') and K gamma (x,y | z,z'), intro- 
duced in $2 and §5, respectively, are related to each other by the transformation 
K i ► K°- £ , where 



e(x) 



1, X £ Z' + 



(-l) fc , x = -{k + \)£7J_, k = 0,1,2,... 



Proof. This follows from Theorem 4.2 if we pass to the limit as £ /" 1. On the other 
hand, this can be readily checked directly, because the crucial step, the coincidence 
of both kernels for x = y £ Tl_ , was already verified in the proof of Theorem 4.2. □ 

The next result, which we have just used in the proof of Theorem 4.2, is also 
of independent interest. It is a generalization of the differentiation formula for the 
discrete Bessel kernel, see Borodin-Okounkov-Olshanski [BOO, (2.11) and below]. 

Proposition 4.5. We have 

^ K(x, y | z, z', = ^ (P(x | z, z', 0Q(y I z, z' , + Q(x \ z, z', $P(y \ z, z' , £)) • 
Proof. This can be directly verified by making use of the differentiation formulas 
| P{x | z, z', = (| - ^±^j P(x | z, z', - ff^ QO* I z, 0, 
| Q{x | z, z> = (- ± + 0^) Q(x | z, z>, + $^P{* I *, A 0. 

To check these formulas we use the following differentiation formulas for the Gauss 
hypergeometric function, which can be derived from [Erl, 2.8 (20), (27)]: 

d f F{a,b;c;^)\ _ ab F(a + 1, b + 1; c + 1; ^) 



<%\ r(c) ; (i-0 2 r(c + i) 

,&;c; A) 

□ 



1 F(a- 1,6- l;c- 1;^) + c -l\ F(a,6;c;^ T ) 



$ r(c-i) V i-£ £ ; r(c) 



5. The projection property 

Let i/ = ff + ® be a Hilbert space decomposed into a direct sum of two 
subspaces. According to this decomposition we will write operators in H in 2 x 2 
block form. Let A : H- — > ii + be a bounded operator and let 



L = 



A 

-A* 



This is a bounded operator in H. Notice that 1 + L is invertible. Indeed this follows 
from the fact that 



(1 + L)*(1 + L) = 



1 + AA* 
1 + A*A 



> 


"1 


0" 







1 
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Set K = L(l + L) 1 and write K in the block form, 

K = 



a b 
c d 



Next, set 



K° = 



a b 
-c l-d 



°K = 



1 - a 
c 



d 



Proposition 5.1. The operators K° and °K as defined above are orthogonal pro- 
jections onto the subspaces 



H° 



{Ah- @h-\h-e H-}, °H = {h+ e {-A*)h+ \ h+ e H+}, 



which are essentially the graphs of the operators A and — A* , respectively. We have 
K° ■ °K = °K ■ K° = and K° + °K = 1. 



Proof. The latter equality is immediate from the definition of K° and °K. 

Obviously, H° and °H are closed subspaces, orthogonal to each other. Moreover, 
as is well known, their sum is the whole H. (Indeed, it suffices to check that any 
/ G H + can be written as a sum of vectors from H° and °H. This means that 

h- - A*h+ = 0, Ah- + h+ = f, 

which is reduced to (1 + AA*)h + = f. But the latter equation is solvable, because 
1 + AA* is invertible.) 

Next, one can directly verify that 

a = (1 + AA*) _1 AA* = AA*(1 + AA*)- 1 

b=(l + AA*y 1 A = A(l + A* Ay 1 
c=-(l + A* A)' 1 A* = -A*(l + AA*)- 1 



d = (1 + A* A)- 1 A* A = A*A(l + A* A) 



-l 



Using these explicit expressions for the blocks a, b, c, d one can readily check that 
the operator K° is the identity on H° and zero on °H. Similarly, the operator °K 
is the identity on°H and zero on H°. This concludes the proof. □ 

Remark 5.2. The claim of Proposition 5.1 remains true under weaker assump- 
tions. Namely, A may be an unbounded, closed operator with dense domain. □ 

Theorem 5.3. The discrete hypergeometric kernel K_(x,y | z,z',£) on 11 x Tl , 
as defined in %2, is a projection kernel. That is, it corresponds to an orthogonal 
projection operator in the Hilbert space £ 2 (Z'). 

Proof. Take H = £ 2 (Z'), H + = £ 2 (Z' + ), H- = £ 2 (Z'_), and let K be the operator in 
H defined by the kernel K (x,y | z,z',£). By Theorem 3.4, K = L(l+L) _1 , where L 
\ A~\ 

has the form ^ with a certain bounded operator A (recall that the kernel 

A(x,y) is real, so that the adjoint operator A* is given by the transposed kernel). 
Let K be the operator given by the kernel K_(x, y | z, z', £). By Theorem 4.2, K_ = 
eK°e- 1 1 where e is a diagonal matrix with ±l's on the diagonal. By Proposition 
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5.1, the operator K° is an orthoprojection. Therefore, K_ is an orthoprojection, 
too. □ 

We would like to prove a similar claim for the gamma kernel JC samma (:c, y | z, z'). 
By Theorem 2.3, it is the pointwise limit (as £ / 1) of the hypergeometric kernels 
K_(x, y | z, z' , £), which are projection kernels by virtue of Theorem 5.3. That is, the 
operator defined by the gamma kernel is a weak limit of orthoprojections. However, 
the projection property is not stable under limit transitions in the weak operator 
topology. Indeed, one can obtain any selfadjoint operator with norm < 1 as a weak 
limit of orthoprojections in an infinite-dimensional Hilbert space. It would be nice 
to strengthen Theorem 2.3 by proving that the kernels (or rather the corresponding 
operators) actually converge in the strong operator topology: this would suffice to 
conclude that the limit kernel inherits the projection property. However, to derive 
the strong convergence directly from the formulas, as we have done for the weak 
convergence, does not seem to be easy. 

Below we present a simple argument, which proves the strong convergence in 
a roundabout way, under an additional restriction on the parameters z,z'. The 
idea is to prove an analog of Theorem 3.4. To do this we verify the strong con- 
vergence of the "L-operators" , whose kernels are much simpler than those of the 
"if-operators" . We will impose a restriction on the parameters z, z' to ensure the 
boundedness of the limit "L-operator" . 

Consider the kernel A(x, y | z, z', £) on Z' + x Z'_ introduced in §3. As £ /" 1, the 
factor £( x -v)/ 2 tends to 1, so that the kernel pointwise converges to the kernel 



A(x,y | z,z') 



-\/sin(7rz) sin(7rz') 



^T{z + x + \)T{z' + x + \) ^T{-z-y+\)T{-z> - y + \) 1 



r(x+i) n-v+k) x -y 

xei' + , y ez'_. 



7 A) 



Proposition 5.4. Assume \z + z'\ < 1. Then the operator A : £ (Z'_) 
with the kernel A{x, y \ z, z') is bounded. 

Furthermore, let A^ stand for the operator with the kernel A{x,y \ z,z',£). As 
£ / 1, we have A^ — > A and A| — > A* in the strong operator topology. 

Recall that (z,z') is subject to one of the conditions (i), (ii) of Proposition 1.8. 
This implies, in particular, that z + z' is real. If (z, z') satisfies condition (i) then 
the additional restriction \z + z'\ < 1 means that z = z' lies in the strip 3?( • ) < 1/2. 

Proof. The second claim easily follows from the first one. Indeed, we have A^ = 
EAE, where S = diag(l, £, ^ 2 , . . . ). Observe that ||S|| = 1, S converges to the 
identity operator in the strong operator topology. Next, we have ||S|| = 1 for any £. 
Since operator multiplication is a jointly strongly continuous operation on bounded 
sets, the strong convergence A^ — > A and A| — » A* follows. 

Let us prove the first claim. Let / and g range over the unit balls of the Hilbert 
spaces £ 2 (Z' + ) and £ 2 (Z'_), respectively. We have 

\\A\\=snp\(Af,g)\< S np\J2 £ \A(x,y \ z, z')\ ■ \f(x)\ ■ \g(y)\ 



f,g f,g 
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By virtue of (3.1), in order to prove that this quantity is finite, it suffices to prove 
that 

- P (.,i£fe)*-^H- |/W| -H <+ - 

ft is convenient to rewrite this as 

-1^)11 ' - 




x + y 

Here we assume that both / and g range over the unit ball of £ 2 (Z' + ). 

Next, we may replace the sums over 11, by the integrals over R + with respect to 
Lebesgue measure (assuming that / and g range over the unit ball of L 2 (R + ,dx)). 
Indeed, this will only strengthen the claim. The resulting claim is equivalent to the 
boundedness of the operator in L 2 (R + , dx) with the kernel 

x 



V J x + y 

It is not hard to show that this integral operator is bounded if and only if \z+z'\ < 1, 
see Olshanski [Oil]. □ 

Similarly to (3.2), using the kernel A(x, y | z, z') we construct another kernel on 
I 1 x U by' 

'0, x > 0, y > 

A(x, y | z, z'), x > 0, y < 

-A(y,x | z,z'), x < 0, y > 

0, x < 0, y < 0. 

The next result is the counterpart of Theorem 3.4. 

Theorem 5.5. Let L : £ 2 {Jj') — > l 2 (Z') be the operator with kernel L(x,y | z,z'). 
Assume \z + z'\ < 1. Then the kernel if gamma (a;,y | z 7 z') is precisely the matrix of 
L 

the operator 



L(x,y | z,z') = < 



l + L 

Proof. Let denote the operator with kernel L(x,y \ z,z',£). We claim that 
L^(l + L^)^ 1 strongly converges to L(l + L)~ x . To check this we use a standard 
argument. Write the formal identity 

(1 + Ls)- 1 (1 + i)- 1 = (1 + L ( )-\L L £ )(l + L)-\ 

Since (1+L|)(l+Lj) > 1 (see the beginning of the section) , we have ||(l+i^) _1 || < 
1. Next, the operators are uniformly bounded and — > L strongly: this follows 
from Proposition 5.4 (here we use the assumption \z + z'\ < 1). Therefore, the 
product in the right-hand side strongly converges to 0. 

Since the kernel of L^(l + L^ 1 is K(x,y | z,z',£) (Theorem 3.4), the latter 
kernel strongly converges to the kernel of L(l + L) _1 . On the other hand, we already 
know (Theorem 3.5) that K(x,y | z,z',£) pointwise converges to i4T gamma (a;, y | 
z, z'). We conclude that if gamma (j;, y \ z, z') is the kernel of L(l + L)-\ □ 
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Theorem 5.6. Assume \z + z'\ < 1. The gamma kernel K_ gamma (x,y \ z,z') on 
Z' x Z', as defined in §2, is a projection kernel. 

Proof. We argue precisely as in the proof of Theorem 5.3, with reference to Theorem 
4.4 instead of Theorem 4.2. □ 

We conjecture that the claim of Theorem 5.6 holds without the restriction \z + 
z'\ < 1. 

6. The tail kernel 



Let us study the asymptotics of the process ps a ™ ma (equivalently, of P| ;a ™ ma ) 
near +oo. In order to find a suitable scaling, let us look at the first correlation 
function (also called the density function) . It is given by the value of the correlation 
kernel on the diagonal, which was written down in Comment 3 to Theorem 2.3 in 
terms of the psi function ip(x). Near +oo, the psi function behaves as follows ([Erl, 
1.19(7)]) 

ip(x) — logx - — + 0(x~ 2 ). 

Substituting this into the expression for _K gamma (x, x \ z, z') we see that the density 
function of the process Pf™ a behaves as 

(z — z') sin(7rz) sin(7rz') , . . 

" ■ -, n L -x~\ x^+^. 6.1 

7rsm(7r(z — z')) 

This suggests that the scaling should have the form x = e so+s , where so — ► +oo, be- 
cause then in the coordinate s, the density function will be asymptotically constant. 
Notice that in the limit transition, the lattice turns into the real line. 

All statements of this section are made under the assumption that (z, z') satisfy 
one of the conditions (i), (ii) of Proposition 1.8. 

Proposition 6.1. In the scaling limit x = e Sa+s , where s — > +oo, the correlation 
functions o/Pj a ™ ma converge, and the limit functions have determinantal form with 
the kernel 

tail sin(7rz)sin(^') sinh(§(z-z')(s-f)) 

K {s,t\z,z) = — — — . rjri iv; > s,teR. 

7rsin(7r(z - z'j) smh(%(s-t)) 

Proof. It suffices to examine the limit behavior of the correlation kernel K_ gamma (x, y | 
z, z'). Recall that if we are given a correlation kernel K(x, y) on a state space with 
reference measure dx then, under a transformation of the state space, we have to 
look at the transformation of the expression yjdxdy K(x, y), rather than of K{x, y) 
itself. In our situation, x = cxp(s +s), y = cxp(s +t), so that y/dxdy — ^fxy\/ dsdt. 
Using the well-known asymptotics of the ratio of gamma functions ([Erl, 1.18 (4)]) 
and the explicit expression of the kernel in question we find that the limit 



lim \j5yK samma (x,y \ z,z') 



^tail 



=cxp(s +s), y=exp(s +*) 



exists and equals K_ (s, t\ z,z'). □ 

We call X tall (s,t | z,z') the tail kernel with parameters z,z'. It determines 
a translationally invariant point process on R. The tail kernel was obtained via a 
double limit transition: first, from the discrete hypergeometric kernel to the gamma 
kernel, and next, from the gamma kernel to the tail kernel. The same result can be 
obtained in one step, as the following proposition shows. 
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Proposition 6.2. Consider the discrete hypergeometric kernel K_(x,y | z,z',£) on 
the lattice U , and make the change of variables x — e So+s , y — e So+t . Let sq — > +00 
and £ f 1. Moreover, assume that (1 — C) -1 grows faster than e s ° ; namely, 

e s ° =O((l-0- E ) 

where e > is small enough (it suffices to assume that e is smaller than 1 — \3t(z — 
z')\; we recall that \di(z — z')\ < 1, see Proposition 1.8). Then the scaling limit of 
the kernel K_(x,y \ z, z',£) is the tail kernel. 

Proof. First, we slightly revise the proof of Theorem 2.3. Specifically, we cannot 
apply the trivial estimate (2.1), because the parameter c = x + \ is no longer 
constant. Instead of this we use the Euler integral representation of the Gauss 
hypergeometric function in the form 

r 1 7;°-! fi — u)- h 

F(a 1 l-c + a;l-b + a;w- 1 )=T(l-b + a) / — { > (1 - uw^Y+^du 

Jo r ( a ) r (! - 0) 

(see [Erl, 2.1.3 (10)]), where, as in Theorem 2.3, (a,b,c;w) is either 

(-z, -z', &+§; ^ OT (~ z + ~ z ' + x + h ■ 

(Notice that in [Erl] the Euler integral representation is given under restrictions 
on the parameters. However, these restrictions are inessential, because, in our 
notation, the expression WT^b) ma kes sense as a distribution supported by 
[0, 1], for any complex a, b.) 

By our hypothesis, (c + a — l)w; _1 = O ((1 — C) 1_e )i whence 

(1 - uw- 1 )^- 1 = 1 + O ((1 - O 1 ^) 

uniformly in u e [0, 1]. This gives 

F(a,l - c + a;l - fe + ajw" 1 ) = 1 + O ((1 - 1_£ ) 

and likewise 

F(b, l-c + b-l-a + b- w- 1 ) = 1 + O ((1 - f ) 1_£ ) . 

Then we may continue the argument as in the proof of Theorem 2.3 and use the 
same simple estimate for ratios of gamma functions as in the proof of Proposition 
6.1. □ 

The gamma kernel and the discrete hypergeometric kernel in the second form 
(which corresponds to looking at the Frobenius coordinates of Young diagrams) 
also have tail limits. 

The density function of ps a J lma has the asymptotics 

(z — z') sin(7rz) sin(7rz') 

r- r7 \x\ , x~>±oo, 

7rsm(7r(z — z')) 

which immediately follows from the asymptotics (6.1) of the density function of 
pgamma indeed, ps a ™ ma an d £^™a coincide on U + , and the change of sign 
transformation of f 2 g ™ ma is equivalent to changing the signs of the parameters 
z, z' . Thus, it makes sense to consider the scaling limit of ps a f ma ^ both pi us anc | 
minus infinity. 
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Proposition 6.3. In the scaling limit x = ±e So+s , where sq — > +00, the correlation 
functions of psamma converge, and the limit functions have determinantal form with 
the kernel given by 

• For x = e Sa+s , y = e So+t , the limit is the same as in Proposition 6.1: 

sin(7rz) sin(7rz') sinh(i(z — z')(s — t j) 
■k sin(-K(z — z')) sinh(i(s — t)) 

• Forx = e so+s , y = -e sa+t : 

y/sm(irz) sin(7rz') sin(-7rz)e^ z_z '^ s_ *' — sin(7rz')e~3( z - z ')(s-*) 
7rsin(7r(z - z')) 2 cosh(|(s - t)) 

• Forx = -e S0+s , y = e So+t : 

^/sin(7rz) sin(7rz') sin(7T2:')e5( z ~ z ')( s ~*) — sin(7rz)e _ 5( z ^ z ')( s -*) 
7rsin(7r(z — z 1 )) 2 cosh(i(s — t)) 

• For x = —e so+s , y = — e sa+t the kernel is the same as for 
x = e So+s , y = e Sa+t (the first case above). 



We denote the resulting tail kernel in the second form by K tall (s,t | z,z'). It 
defines a determinantal point process on Rxl which is invariant under simultaneous 
translations (s, t) i — ^ (s + A, t + A), Ae M. This kernel appeared for the first time 
in [Oil, Proposition 4.1], see also [BOl, Theorem VII]. 

Proof of Proposition 6.3. The formulas for K tall (s,t | z,z') are readily obtained 
from those for K sammli (x, y \ z, z'), see Theorem 3.5, using the standard asymptotics 
of ratios of gamma-functions, see [Erl, 1.18 (4)]. One also has to keep in mind the 
transformation of differentials explained in the proof of Proposition 6.1. □ 

The statement of Proposition 6.2 also carries over. 

Proposition 6.4. In the scaling limit x = ±e So+s , where s — > +oo with 

e s « =0((l-e)" e ), 0<e<l-|R(*-*% 

the discrete hypergeometric kernel K(x,y | z,z',£) converges, as £ /" 1, to the tail 
kernel K tail (s,t \ z,z'). 

The proof goes along the same lines as that of Proposition 6.2, and we omit it. 

Proposition 6.3 and 6.4 prove the convergence of the correlation kernels. In many 
situations it is simpler to establish the corresponding convergence of L-kernels (as 
usual, L = K(l — K) -1 , where K is a correlation kernel). We have already used 
the convergence of L-kernels, see Theorems 5.5 and 5.6. The following statement 
shows that the convergence of the discrete hypergeometric kernel and the gamma 
kernel to the tail kernel can be seen on the level of the corresponding L-kernels. 

Proposition 6.5. In the scaling limits of Propositions 6.3, 6.4, the kernels L(x, y | 
z,z') defined at the end of $5, and L(x,y | z,z',£) defined by (3.2) converge to a 
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kernel on R x R given by 



For x = e So+s , y = e s " +t , the kernel is identically equal to 



Forx = e sa+s , y = -e So+t : 



y/sm(Trz) sin(7rz') e l( z + z )( s *) 



• For x 

• For x 



-e So+s , y = e So+t : 



7T 2cosh(|(s-f)) 
^Jsvc\{itz) sin(7rz') e -?( z + z ')( s ~ t ) 



2cosh(±(s-t)) 



-e so+s ,y 



O so+t 



, the kernel is also identically equal to 0. 



Proof. Direct computation. □ 

Let us denote the kernel denned in Proposition 6.5 by L tall (s, t). It is easy to see 
that it defines a bounded operator in L 2 (RUR) if and only if \z + z'\ < 1. Similarly 
to Theorems 3.4 and 5.5, we have the following claim. 

Proposition 6.6 [Oil, Proposition 4.2]. If \z + z'\ < 1 then 



K 



tail 



£tail 

1 + L tail ' 



Proof. Let us identify L 2 (M U R) with L 2 (R) L 2 (R). Then we may interpret 
integral operators in this Hilbert space as 2 x 2 matrix-valued integral operators 
on R. Thus, we may write 



^tail 



r tail t tail 

■^11 "^12 

j tail r tail 

^21 ^22 



K 



tail 



retail retail 

^11 ^12 

I^tail rtail 

A 21 -^22 



where all the blocks are integral operators in L 2 (R), 

L\f = Llf(s,t), K*f = K*f{s,t), i,j = 1,2. 
Actually, these integral operators are translationally invariant, so that we may write 
Llf(s,t)=Llf(s-t), K*f(s,t) = K$f(s-t), i,j = 1,2. 

Given a function f(s) on R, let f(u) denote its Fourier transform, 



/ + OC 
< 
-OO 



e ms f(s)ds. 



The Fourier transform is an isometry between L 2 (R, ds) and L 2 



271 



). By virtue 



of the translation invariance, the Fourier images of L tal1 and K tal1 are operators of 
multiplication by 2 x 2 matrix-valued functions in u: 



L tail (w) 



L\f{u) L\f(u) 
L\f{u) P 2 f(u) 



K tai \u) 



K\f{u) K\f{u) 
klf(u) L$$ l (u) 
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From the explicit expressions for L and K (see Propositions 6.3 and 6.5) it follows 
that their Fourier images have the form 



L tai \u) 



c(u) 

~4uj o 



K tai \u) 



a(u) b{u) 
—b{u) a(u) 



where 



c(u) 



a(u) 



!^so\{ttz) sin(7rz') e l( z + z ') s 1 
n 2cosh(f)/ s 

sin(7rz) sin(7rz') sinh(i (z - z')s) 



b(u) 



_7rsin(7r(z- z')) 2sinh(§) J ^ 

^/sin(7rz) sin(-7rz') sin(7rz)e^ z_z '' s — sm(Trz')e~^ z ~ z '^ s 
7rsin(7r(z — z')) 



2cosh(f) 



The required Fourier images can be evaluated from the tables, see formulas 
1.9(14) and 3.2(15) in Erdelyi [Er2], 



f sinh(|(z - z')s) 
\ sinh(is) 



r - 



27rsin(7r(z — z')) 
cos(2iriu) + cos(7r(z — z')) 



[2cosh(§) 

From these formulas we get explicit expressions 

c(u) = 
a{u) 



cos(niu — \k{z ± z 1 )) 



\J sin(7T2;) sin(7rz') 
cos(7rm — \k{z + z 1 )) 
2 sin(7T2;) sin(7rz') 



b(u) = 2^sin(TTz) sin( 



cos(2-7riu) + cos(-7r(z — z')) 

cos(7riu + \k{z + z'j) 



cos(27riu) + cos(7r(z — z')) 
Now, the claim of the proposition is equivalent to the relations 

c(u) 



a(u) = 



c(u)c(u) 



b(u) = 



1 + c(u)c(u) 1 + c(u)c{u) 

which are checked directly from the above expressions. □ 

7. ZW-MEASURES ON SIGNATURES 

In this section, we replace the set Y of Young diagrams by the set SGN(iV) of 
signatures of length N. Here N — 1,2,..., and a signature A e SGN(iV) is an 
iV-tuple of weakly decreasing integers, 



A = (Ai, . . . , A;v), Ai > • • • > Ajv • 
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We will describe a family of probability measures on the sets SGN(TV) (for more 
detail, sec Olshanski [012] and Borodin-Olshanski [B04]). Then we will study the 
behavior of the measures as N — > oo, there the limit transition is similar to the 
"second regime" considered in §2 above. We show that the final result is again 
described in terms of the gamma kernel. 

Our probability measures on SGN(iV) depend on 4 complex parameters z, z', w, w' 
and have the form 



M z ^^ jW ,\ N (\) = (constAr) 1 • M' ZZ , WW ,\ N {\) 



where 



N 



m 'z,z',w,w'\nw - n 



Li \^-Xi + i)T(z'-\i + i) 

1 



' T(w + N + 1 + A, - i)T{w' + N + 1 + A, - i) 

Xi - Xj+j -i 



(Dim N (X)) 2 , 



Dim N (X)= {'I 



1 — 1 

l<i<j<N J 



and 

COnstAT = M 'z,z',w,w'\n( X ) 
AGSGN(AT) 

is the normalizing constant depending on z, z' ,w,w' , N . Under suitable condi- 
tions on the quadruple (z, z', w, w'), the measures M ZiZ i iWiW '\n are well defined for 
all N. That is, the weights M' z z , w W ,\ N {^) are nonnegative and their sum over 
A e SGN(iV) is finite. A criterion for that to happen and for all the weights 
M z z i tW W i\ N {X) to be strictly positive is provided below. See [012, §7] for detailed 
explanations. 

The measures M z z i tWjW '\N can be obtained by a construction which is quite 
similar to that described in §1, with the finite symmetric group S n replaced by the 
compact group U(N) of unitary TV x N matrices. Let be the normalized Haar 
measure on U(N), and let Hn be the Hilbert space of square integrable functions 
on U(N) (with respect to [In), constant on conjugacy classes. In Hn, there is 
a distinguished orthonormal basis formed by the irreducible characters xa of the 
group U(N). Here A ranges over SGN(JV). 

Let T be the unit circle in C and T N be the product of N copies of T (the 
iV-dimensional torus). Given a unitary matrix U € U(N), we assign to it the 
unordered iV-tuplc of its eigenvalues, {u\ 1 . . . , ujv). Any element of Hn can be 
viewed as a function in (ui, . . . , uat), that is, as a symmetric function on the torus 
T^. In particular, the irreducible characters Xa(^) are the (rational) Schur func- 
tions s x (u 1} . . .,u N ), 

det kt, 

l<i,j<N L 

S A («1, ■ • • ,U N ) = ; N _ . 



det \uy 

l<i,j<N 1 
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The whole Hilbert space Hn can be identified with the Hilbert space of symmetric 
functions on T N , square integrable with respect to the measure 

1 N 
fj, N (du) = — Y[ lui-uj^Y^dut, 

l<i<j<N i=l 

which is the push-forward of under the correspondence t/ i — > (ui, . . . , u N ). Here 
dui is the normalized invariant measure on the zth copy of T. 

Given two complex numbers z,w, we define a symmetric function on T N by 

JV 



If 3?(z + w) > — \ then J z . w \n belongs to the space Hn- Let (z',w') be another 
couple of complex numbers with 3?(z' + w') > —\- We set 

/w {fz,U)\N> XA)(XA) f^JT TT\n) . r,^1^T/7»r\ 

Mz, Z ',to,to'|Ar(A) = r ' , A G SGN(JV), 

\Jz,w\Ni J w ' y z '\n) 

where ( • , • ) is the inner product in Hn • It turns out that this definition leads us 
to the explicit formula given above. Notice that Dim^ A is equal to the value of 
the character xa at 1 e U(N) (equivalently, to the value of the Schur function s\ 
at (1,...,1) G T N ). 

Similarly to the identification of the Young diagrams with points in {0, 1} Z 
described at the beginning of §2, we identify a signature A = (Ai, . . . , \n) with a 
binary sequence X(X) = (..., a_ 3 / 2 , a_i/ 2 | a 1/2 , a 3/2 . . . ) by 

l, ifje{\i-i + ±\i = i,...,N}, 

0, otherwise. 

Note that this identification establishes a one-to-one correspondence between SGN(JV) 
and the elements from {0, 1} Z with exactly N l's. 

In what follows we will assume that neither of the parameters z,z',w,w' is 
an integer. This is always the case if we require the weights of all signatures 
to be nonzero. Further, the condition of M Z Z> W ^ W ,\ N (X) of being positive for 
all A e SGN(iV) is equivalent to both pairs (z, z') and (w, w') satisfying one of 
the conditions (i) and (ii) of Proposition 1.8, and the convergence of the series 
12\eSGN(N) M z,z>,w,w>\nW is equivalent to the inequality 

3?(z + z' + w + w') > -1, 

see [012] for proofs. We also assume these conditions to be satisfied. 
The following statement is an analog of Theorem 2.2. 

Theorem 7.1 ([B04, Theorem 7.1]). Let P_ Z Z , W W ,^ N be the push-forward of the 

measure M z z i w w :\ N under the embedding A 2L.W of SGN(TV) into {0, 1} Z 
defined above. Then its correlations functions have determinantal form 

p m (xi, ■ ■ ■ ,X m I P, z > WiW n N ) = det \K_{Xi,Xj | z,z',w,w' | N)], 

m = 1,2, . . . , xi,.. .,x m € Z ; , 
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where the correlation kernel is given by 

1 Pn(x)pn-i(v) - p N -i(x)p N (y) 



K(x,y 



z.z ,w,w 



Pn(x) 



I N ) , 
T(x + w' + N + i) 



pjv-i(a;) 

/ijv-i = 
/(*) = 



r(a: + io' + 
r(a; + U ;' + iV+ i) 



r(a; 



1) 



3^2 



3^2 



x- y 

-TV, z + w', z' + w' 
E, x + w' + 2 



-N + 1, z + w' + 1, z' + w' + 1 



2, £ 



+ 1 



r 



iV, S + 1, S + 2 
£ + TV + 1, ^ + tu + 1, z + w' + 1, 2;' + to + 1, z 1 + w' + 1 

1 



r (z - x + 1) r (z' - 1 + 1) r (w + x + n + i) r (w' + x + n + 1) • 

urcf/i £ = z + z' + w + w' . 

Notation. The symbol 3F2 above stands for the higher hypergeometric series of 
type (3,2), see e.g. [Erl, chapter 4] and [Ba]. We also use the notation 



a, b, 
c, d, 



r(a)r(b)- 

T(c)T(d) ■ 



Comments. 1. The functions Pn~i and p^ are monic (i.e., the highest coefficient 
is equal to 1) orthogonal polynomials on Z' of degree (N — 1) and N, correspond- 
ing to the weight function f(x), and /ijv-i = Wpn-iW^^i jy The determinantal 
structure of the correlation functions with the kernel expressed through orthogonal 
polynomials as above is a standard fact from Random Matrix Theory. Up to the 
factor a/ f(x)f(y), the kernel is the Christoffel-Darboux kernel for the orthogonal 
polynomials with weight f(x). 

2. If S = then the formula for pn above does not make sense because it 
involves a hypergeometric function with a zero lower index. However, the kernel 
itself admits an analytic continuation to the set S = 0, see [B04, (7.3)]. 

The next statement is an analog of Theorem 2.3. 



Theorem 7.2. The measures P 



z.z' ,w.w 



n N weakly converge, as N — > 00, to the 



probability measure ps a ™™^ on {(^ 1} Z defined in Theorem 2.3. A 

Proof. Similarly to the proof of Theorem 2.3, we will show that the kernel K_ (x, y \ 
z,z',w,w' I N) has a pointwise limit as N — > 00 which equals the gamma kernel. 
We will assume that x 7^ y, z ^ z', and E ^ 0. The convergence is easily extended 
to these sets by analytic continuation, as is explained at the end of the proof of 
Theorem 2.3. 

We will use the following transformation formula for 3 F 2 with the unit argument, 
see Bailey [Ba, 3.2(2)]: 



3^2 



a, 6, c 
e,f 



1- a, e, f, c-b 
b, f — b, 1 + b — a, c 



3^2 



b, b — e 
1 + 6- 



-1, 6-/ + 1 

c, 1 + b — a 



+ a similar expression with b and c interchanged. 



4 In fact, Theorem 2.3 provides formulas for the correlation functions of ps 8 ™™^ 
uniquely define the measure. 
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If a — > — oo, and b,c,e,f are fixed, the 3F 2 's in the right-hand side are equal to 
1 + 0(|a| _1 ), and using 



r(i - a) 

T(l + b-a) 
we obtain 



(-a)- b (l + Odal" 1 )), r( 1 j ( | c " ) a) = (-a)- c (l + 0(\a\-')), 



3 r 2 



a, b, c 
e, / 



(-a)- b r 



e, f,c-b 
e -b, f - b, c 

+ (-a)- c r 



(l + Odal" 1 )) 

e, f,b-c 
e-c, f -c,b 



(1 + Oflar 1 )). 



Applying this estimate to pat and pjv-i with a = — AT and —N + 1, respectively, we 
get 



P;v(a:) = ( 



E, z'-z 
z' + w, —z + x + i, z' + to' 



(l + <>(*)) 



+ a similar expression with z and z' interchanged J • T (x + w' + N + |) . 

£ + 2, z' - z 
l 

2' 

+ a similar expression with z and z' interchanged J ■ T (x + w' + N + |) . 



(*)=( 



Z' + W + 1, -Z + X + ± z' + w' + 1 



As we substitute these formulas into the expression p^{x)pN-i{y) — Pn-i(x)pn(v), 
we see that the part coming from 0(]y) is equal to 



r (x + w' + N + i) T (y + + N + ±) JV 



-2-z / -2to'-l 



•0(1), 



due to the fact that |3?(z-z')| < 1 and A^-^O^) = o(l) as N -» oo. Further- 
more, four of the remaining eight terms cancel out, and we get (using the relation 
T(s + 1) = sT(s) a few times) 



p N (x)p N - 1 (y) - p N -i{x)p N (y) =r(x + w' + N+±)T(y + w' + N+±) 

S, £ + 2, z' - z, z - z' 
z + w, z' + w, z + u/, z' + w' 

1 



x jy-z-z' -2u/-lp 



r(-z + a; + |)r(-z' + y + ±) \ (2 + w){z + w') {z> + w)(z' + w') 



1 



1 



1 



T(-z' + x + \)T(-z + y+\) \{z' + w)(z> + w') (z + w)(z + w') 
Simplifying 

1 1 (*'-*)£ 



o(l) 



(z + w)(z + w') (z' + w)(z' + w') (z + w)(z' + w)(z + w')(z' + w') 
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and using the formula for /ijv-i, we see that 

Pn{x)pn-i{v) - Pn-i(x)pn(v) T(S + N + 1) 



((z'-z)T(z'-z)r(z-z')) 



h N -i{x-y) T(N) 
x T (x + w' + N + I) r (y + w' + N + i) jy-*-*'^™'-^! + o(l)) 
1 1 \ 1 



T{-z + x+\)T{-z' + y+\) T(-z' + x + \)T{-z + y + \) J x-y 

Since T(S + N + 1)/T(N) ~ N^ +1 and (z' - z)T(z' - z)T(z - z') = tt/ sm(ir(z- z')), 
we see that the above expression equals 

— T (x + w' + N + I) r (y + w' + N + ±) N w - W \l + o(l)) 



sin(-7r(z — z')) 

1 1 \ 1 



Y{-z + x+ \)T(-z' + y+ \) T(-z' + x + \)T{-z + y + \) J x-y 



It remains to multiply this expression by y/ f(x)f(y) and take the limit N — > oo. 
The weight function f(x) consists of four gamma-factors, two of which do not 
depend on N while the two others do. Taking the factors in \J f{x)f(y) which are 
independent on TV, we obtain 



^r(z - x + i)r(z' - x + i)r(z - y + i)r(z' - v + \) 



T{-z + x + \)T(-z' + y+\) T(-z' + x + \)T(-z + y + ±) 
sin(7rz) sin(7rz') 



T(-z + x+ \)Y(-z' + x+ i)r(-z + y + \)T{-z' + y + i) 
x (T(-z' + x+ i)r(-z + y + i) - T(-z + x + \)T{-z' + y+\)). 

Here we used the fact that due to our restrictions on (z, z'), the product sin(7rz) sin(7rz') 
is always positive, so we can pull it out of the square root. 

As for the gamma-factors in \/.f{x)f(y) that do depend on N, we get 

r (a; + w' + N + i) T (y + w' + N + ±) N w ~ w ' 
yjr(w + x + N+ ±)T(w' + x + N+ ±)T(w + y + N+ i)r(w' + y + N + i) 

= i + o(£). 

Thus, gathering all pieces together, we see that as N — > oo we have the estimate 

K(x, y | z, z', W , W > | N) = V^) PN -^)-V N -^v) 

Hn-i(x — y) 

= KZ—(x iy \-z,-z')- (l + o(l)). □ 
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Remark 7.3. Observe that the measure M ZZ ^ W ^ W ,\ N has the following symmetry 
property: 

M ZtZ i iWfW >\ N (\i, . . . , Xn) = M w ^ w i jZjZ ,\ n (—\ n , . . . , — Ai). 

Hence, the measure P_ z z , w w n N on {0, 1} Z is invariant with respect to the simul- 
taneous switch (z, z') < — > (w, w') of the parameters and the involution 

{0, 1} Z ' -» {0, 1} Z ', (flj)jez' ^ («j = a- N -j) jeZl . 
This means that Theorem 7.2 also implies the following claim: Embed SGN(TV) 
into {0, 1} Z ' = {(..., a-3/2, «-i/2 I a 1/2 , a 3 /2, • ■ • )} by 

r i, if j e{-\i-(N + i-i) + ±\ i = i,...,N}, 

Clj = < 

{ 0, otherwise. 

Then the push-forwards of the measures M z z * w w i\ N under these embeddings 
weakly converge to P^^^w' as N — > oo. 

Remark 7.4. It is natural to ask whether the two limit transitions, the one of 
Theorem 7.2 and the one described in Remark 7.3 above, lead to asymptotically 
independent random point processes. The answer turns out to be positive, and the 
exact statement is as follows. 

Consider an embedding of SGN(iV) into {0, 1} Z ' x {0, 1} Z ' defined by using the 
map A i ► X(\) (described just before Theorem 7.1) on the first coordinate, and 
using the map described in Remark 7.3 on the second coordinate. Then the push 
forwards of the measures M ZiZ > tWtW i\jq under these embeddings converge, as N — > oo, 
to the product measure P^™™ ® P. 6 ^™™^' ■ 

The proof follows from the fact that the correlation kernel K_(x, y \ z, z', w, w' \ 
N) tends to zero as TV — > oo if one of the arguments (a;, y) is in a finite neighborhood 
of 0, while the other one is in a finite neighborhood of — N. To prove such an 
estimate one uses the symmetry of the polynomials pn (x) and Pn-i(x) with respect 
to (z, z', w, w', x) < — ► (w, w', z, z', N—x) (which follows from the obvious symmetry 
of the weight function f(x)), and the same estimate of the 3F2 series as was used 
in the proof of Theorem 7.2 above. 

Similarly to the discrete hypergeometric kernel, the kernel K_(x, y | z, z', w, w' \ 
N) of Theorem 7.1 also has a second form. This form corresponds to a repre- 
sentation of the signatures A <G SGN(A f ) through Frobenius coordinates. Given 
a signature A £ SGN(iV) we view it as a pair of Young diagrams (A+, A~): one 
consists of positive Aj's and the other one consists of minus negative A,'s, zeros can 
go in cither of the two: 

A = (A+, Aj , • • • , -A 2 , -A 1 ). 

Write the diagrams A + and A~ through their Frobenius coordinates: 

A* = {pf,...,p% I qf,...,q± ± ). 

Now we associate to the signature A a finite subset X(\) C Z' (or, equivalently, an 
element in {0, 1} Z ) as follows: 

= {Pt + \] U {-<?+ -\}U {-pj -N-±}u{qr-N+ I}, 

where i — 1, . . . , d + and j = 1, . . . , d~ . Then we have the following analog of 
Proposition 3.1. 
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Proposition 7.5 ([B04, Proposition 4.1]). For any A e SGN(iV), the two finite 
subsets 2£(A) and X(X) are related by 



X(X) = X(X)A {-I, -|, . . . , -N + |} , X(X) = X(X)A {-§, -f , . . . , -N + \ } , 



w/iere A denotes the symmetric difference of two sets. 

Note that the notation in [B04] is slightly different, all points are shifted to the 
right by N/2 comparing to our notation here. 

Theorem 8.7 of [B04] proves that the push-forward of the measure M z z , tW W ,\ N 
under the map A i— > Al (A) has determinantal correlation functions and gives explicit 
formulas for the kernel. Let us denote the correlation kernel by K(x, y | z, z', w, w' | 
N). (Once again, this kernel is different from that in [B04] by the shift x i— ► x+ 

The two correlation kernels, K_{x, y \ z, z' , w, w' \ N) of [B04, Theorem 7.1] used 
in Theorems 7.1, 7.2 above, and K(x,y | z,z',w,w' | N) are related by a simple 
transform similar to that of Theorems 4.2 and 4.4. This fact is explained in [B04, 
Theorem 5.10] in a fairly general framework. Together with Theorems 4.4 and 7.2, 
this implies that K(x,y | z,z',w,w' \ N) converges to if gamma (x, y \ —z,—z') as 
N —> oo. 

Note that, similarly to Theorem 5.3, the kernel K_{x,y \ z,z\w,w' \ N) rep- 
resents an orthogonal projection operator by the very definition; its range is the 
TV-dimensional space Span{y / /(x), X\J f(x), . . . , x N ~ x ^/ f(x)}. 

Furthermore, [B04, Theorem 8.7] shows that K(x,y \ z,z',w,w' \ N) has a 
rather simple L-kernel, L = K(l — K)~ x , presented in [B04, §6]. (This is an 
analog of Theorem 3.4 above.) Using the explicit form of this L-kernel, it is not 
hard to show that a natural analog of Proposition 6.5 holds true. However, this is 
not enough to ensure the convergence of the correlation kernels for all admissible 
values of parameters (the reason being the unboundedncss of the limit L-kernels 
for \z + z'\ > 1). Hence, it is of interest to evaluate the asymptotic behavior of the 
correlation kernel directly, so we state this as 

Problem 7.6 (cf. Proposition 6.4). Show that in the scaling limit x = ±e So+s , 
where sq — » +oo with 



the kernel K(x,y \ z,z',w,w' \ N) converges to the tail kernel K tall (s,t \ —z,—z'). 

Due to the symmetry explained in Remark 7.3, solving this problem will also 
imply the convergence of K(x, y \ z, z', w, w' \ N) in the scaling limit x = —N^fe s " +S 
to the tail kernel K tail (s,t \ -w,-w'). 



In this section, we deal with the subset SGN + (7V) c SGN(TV) formed by the 
signatures A <E SGN(iV) with X N > 0. Elements of SGN + (7V) may be called non- 
negative signatures of length N. We will consider a family of probability measures 
on SGN + (iV) depending on parameters z, z', a, 6, where (z, z') is a couple of com- 
plex numbers satisfying suitable conditions, and a, b are real numbers such that 
a > — 1, b > — 1. It is convenient to denote 



e S0 = 0(N e ), 



< e < I - \$t(z - z')\ 



8. Z-MEASURES ON NONNEGATIVE SIGNATURES 



S = 



a + b+1 
2 
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We set 

M z , z ,, aMN (\) = (const^)- 1 • M' z zl aMN (X), A e SGN+(iV), 

where 

■fr/ (TV + £ + A, - z)r(A + 2e + A, - i)r(JV + a + 1 + A, t - i) 
*.*'.^\^ ) 11 ^ T(N + b+l + Xi-i)r{N+l + Xi-i) 

x ! ^ 

T(z - A, + i)T(z' - A, + i)r(z + 2iV + 2e + A, - i)r(z' + 2A + 2e + h - i) J 
x II ((N + K-i + sf -(N + Xj-j + s) 2 ) 2 

l<i<j<N 



and 

COnst W = M 'z,z',a,b\N( X )- 
AeSGN+(JV) 

One sufficient condition ensuring the existence of the probability measures for 
all N is z' = z, ftz > -i±^. 

Once again, the above formula can be obtained following the same general 
scheme. As the Hilbert space H N we now take the space of symmetric functions on 
the TV-dimensional cube [—1,1]^, square integrable with respect to the measure 

N 

p, N (dx) = jf[ (xi - Xj) 2 ■ - Xi) a {\ +Xi) h ■ dxi . . . dx n . 

l<i<j<N i=l 



A distinguished orthogonal basis in Hn is formed by the multivariate Jacobi 
polynomials 

dct \P^\ N Ax,)] 
det [P N '_AXi)\ 

l<i,j<N " 3 

where Pm b (y) are the classical Jacobi polynomials, orthogonal on the segment —1 < 
y < 1 with the weight function (1 — y) a (l + y) h . Let us set 

X?V ^) = %i^ . A e SGN + (A^), 

Ir a|jvII 

where || • || stands for the norm in H^. The normalized polynomials X\ b form an 
orthonormal basis in Hn- 
We define 

N 

Xi 

i=l 

Then the formula for the measure is obtained from the expression 



f z \ N (xi, ■ ■ -,x N ) = 



M ZtZ ,, aMN {\) = {fz]N,Xx ){Xx J ^ n) , X e SGN+(A). 

(Jz\N, J—\n) 
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Notice that for special values of (a, b), the multivariate Jacobi polynomials f^'jjp 
suitably renormalized, can be interpreted as the irreducible characters of the com- 
pact classical groups 0(2N + 1), Sp(2N), 0(2N), or as indecomposable spherical 
functions on the complex Grassmannians U(2N + k)/U(N + k) x U(N). See, e.g., 
Okounkov-Olshanski [OkOl], Berezin-Karpelevich [BK]. 

Let us take the same embedding SGN(iV) into {0, 1} Z (which is identified with 
subsets of 7U) as we took in §7: A X_(ty = {A^ — i + \}^ =1 - Denote by P_ z , z ',aMN 
the push-forward of the measure M zz i ab \^ under this embedding. Standard tools 
of Random Matrix Theory provide us with the following claim, cf. Theorem 2.2 
and Theorem 7.1. 

Proposition 8.1. The correlation functions of the measure P_ z z , a b \ N have deter- 
minantal form 

p m {x 1 ,...,x m \P z , m )= det \K(xi,Xj \ z,z',a,b\ N)], 

1 1 < i, j <m 

m = l,2,..., x 1 ,...,x m &Z', 
where the correlation kernel has the form 

qN(x 2 )qN-i{y 2 ) - qN-i{x 2 )q N (y 2 ) 



K_(x, y | z, z', a,b\ N) 



h N -i ■ (x 2 - y 1 ) 



^/g{x)g{y), 



where x — N + x + e — \, y = N + y + e — |, {qi}i>o are monic polynomials, 
dcgqi = I, satisfying 

^2 1k{x 2 )qi(x 2 )g(x) = h k S kt 

xez 1 



with 



g{x) = {N + e + x-\)T 



xr 



N + 2e + x - ±, N + a + x + \ 
iV + 6 + a;+ i, N + x+\ 

1 

z - x + |, z' - x + |, z + 2N + 2e + x - ±, z' + 2N + 2s + x - \ 



Note that the weight function g(x) vanishes when x < —(N + |) due to T(N + 
x + 5 ) in the denominator. Also note that ^(a;) has a polynomial asymptotics as 
x — > +00, namely 

g{x)^x l - m - 2b - 2 ^ +z '\ x^+oo. 

We will make the assumption that the AN moment of g(x) is finite, which will 
guarantee the existence of qi up to I = N. This means that z + z' > 1 — b. 
The measure P_ z z , a b i N exists under a milder condition of finiteness of the 4(A^— 1) 
moment of g(x), and our results can be extended to this wider domain of parameters 
by analytic continuation. However, we will not provide a detailed argument in this 
paper. 

The weight function g{x) generalizes that associated with the classical Racah 
polynomials, see e.g. [KS]. Namely, if we assume that g{x) vanishes if x is greater 
than some fixed number, which may be achieved by requiring one of the parameters 
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z,z' to be an integer, then {q{\ are exactly the (normalized) Racah polynomials. 
However, it is not immediately obvious how to generalize the Racah polynomials 
to the nonintegral values of z and z' . 

Fortunately, the orthogonal polynomials that we need were recently computed 
by Neretin [Ner] . Actually, Neretin considers even more general situation when the 
lattice is infinite at both plus and minus infinity. Let us state his result. 

Take arbitrary complex numbers ai,a 2 ,a 3 ,a 4 and a, and consider the weight 
function 

a + t 

w{t\a 1 ,a 2 ,a 3 ,a i ;a) = 4 ■ — — -, t e Z. 

llj=i r ( ft j + & + t)T(aj -a-t) 

Proposition 8.2 ([Ner, §3.4]). The polynomials 
Qn((t + a) 2 )=T 



X4F3 



- a\ — a 2 + n, 2 — a\ — a 3 + n, 2 — ai — 04 + n 
2 — a\ — a 2 , 2 — ai — as, 2 — a\ — 04 

n, n + 3 — ai — a 2 — a 3 — a 4 , f — ai + t + a, 1 — a\ — t — a 
2 — ai — a 2 , 2 — ai — a 3 , 2 — ai — 04 



are orthogonal with respect to the weight w(t), and 

sin(27ra) Ilij=i sm (7r(ai + aj)) 



£_^-*n\\- • j / 27T 6 sin(-7r(ai + a 2 + a 3 + a 4 )) 



« ! ntj=i r ( 2 - a i - a J + n ) 



(3 — ai — a 2 — a 3 — 04 + n)r(3 — ai — a 2 — a 3 — 04 + n) 

These statements hold whenever the corresponding series are convergent. 

Note that the polynomials Q n are not monic, the highest coefficient k n of Q n is 
equal to 

T(2n + 3 - ai - a 2 - a 3 - a 4 ) 

k« = (n + 3 - 01 - a 2 - a 3 - a 4 )„ = — — — . 

1 (n + 3 — a\ — a 2 — a 3 — a^) 

In what follows we will use Proposition 8.2 to evaluate the correlation kernel 
from Proposition 8.1 in terms of hypergeometric functions in order to prove the 
following result, cf. Theorems 2.3, 7.2. 

Theorem 8.3. The measures P_ z z , a b i N weakly converge, as N — > 00, to the prob- 
ability measure p^^ 1 ™^ on {0, 1} Z defined in Theorem 2.3. 

Proof. The argument resembles those in the proofs of Theorem 2.3 and 7.1 but it is 
more technically involved. Once again, we will compute the pointwise asymptotics 
of the correlation kernel and show that it converges to the gamma kernel. The 
argument below requires that x ^ y and z 7^ z' . The result is extended to these 
exceptional sets by analytic continuation as explained at the end of the proof of 
Theorem 2.3. 

First of all, applying the identities 

7T 



T(N + 2e + x- ±) - - - - f , 



sin(7r(7V + 2e + x - \))Y{-N - 2e - x + §) ' 

TT 

L ' ' ~ sin(7r(7V + a + x + ±))T(-N - a - x + \) ' 



T{N + a + x+\) 
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we observe that the weight function g(x) of Proposition 8.1 is proportional to the 
weight function w(t | a\, a 2 , a 3 , 04; a) with the following identification of parame- 
ters: 



ai = 1 



t = N + x 
a 2 = b + 1 — e 



2 ' 
a 3 = z 



a = £, 

+ N + £, 



N + £. 



Hence, using this identification and the notation x = N+x+e — \, y = N + y+e—^, 
we may rewrite the correlation kernel K_(x, y \ z, z' , a,b\ N) in the form 



-1 Qn{x 2 )Qn-i{v 2 ) - Qn-i{x 2 )Qn{v 2 ) 



k N H. 



N-l 



(x 2 - y 2 ) 



y/w(x- e)w(y- e). 



It is the asymptotics of this expression that we are going to compute. 

Our next goal is to transform the 4F3 hypergeometric functions that enter the 
formulas for Qn-i and Qn into a form suitable for the limit transition N — > 00. 
We will do this in two steps. 

First, we use the formula [Ba, 7.2(1)]: 



1-F3 



X, Y, Z, - 
U, V, w 



V-Z + n,W-Z + n,V,W 
V - Z,W - Z,V + n,W + n 



x 4^3 



U — X, U — Y, Z, —n 
V + Z-n,l-W + Z-n,U 



which holds if the 4-F3 series are terminating (n = 1,2,...) and Saalschutzian, that 
is, the sum of upper indices is greater than the sum of the lower indices by one: 
U + V + W = X + Y + Z-n+l. 

Applying this formula to Qn with n = N and 

X = N + 3 - a x - a 2 - a 3 - a 4 = 1 - (z + z' + N + b), 
Y=l-a 1 -t-a = -N - x + \ , 
Z=l-a 1 +t + a = N + 2£ + x-j = N + a + b + x+\ 
U = 2 — ai — oi2 = a + 1, 
V = 2-a 1 -a 3 = l-z-N, W = 2 - ai - a 4 = 1 - z' - N, 



we obtain 

Qn{x 2 ) 

X4F3 
Similarly, 



2N + z + a + b + x+l, 2N + z' + a + b + x + \, N + a+ l 
N + z + a + b + x + T;,N + z' + a + b + x+±, a+1 
N + z + z' + a + b, N + a + x+l, N + a + b + x+^, -N 
N + z + a + b + x+ |, N + z' + a + b + x+ \, a+ 1 



Q N - 1 {x 2 ) = r 



2N + z + a + b- 
N+z+a+b+x 



x + i, 2N + z' + a + b + x + \, N + a 



1 + 1, N + z' + a + b- 



\ + 1, a + 1 



X 4 ^3 



N + z + z' + a + b+l, N + a + x+l, N + a + b + x+\, -N+l 
N + z + a + b + x+^ + 1, N + z' + a + b + x+^ + l, a+1 
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The second transformation formula for 4 F 3 that we are about to use looks as follows 
'X, Y, Z, -n 



1F3 



U, V, w 



1 



= r 



X4F3 



l + X-U,l + Y-U,l + Z-U,l-n-U,V,V-W 
V-X,V-Y,V-Z,V + N,l-U,l-U + W 

W-X,W-Y,W-Z,W + n 
1-U + W, 1-V + W, w 



+ a similar expression with V and W interchanged . 

This formula also holds for a terminating Saalschutzian 4F3 series, and it can be 
obtained by successful applications of [Ba, 7.1(1)] and [Ba, 7.5(3)]. 
For Qn, we take n — N and 

X = N + z + z' + a + b 1 Y = N + a + . x + \, Z = N + a + b + x+\, 
U = a+1, V = N + z + a + b + x+\, W = N + z' + a + b + x + \. 

Then the transformation formula yields 



Qn(x 2 ) 



2N + z + a + b + x+±,2N + z' + a + b + x + ±,N + a+ l 
N + z + a + b + x+\, N + z' + a + b + x+\, a+1 



xr 



N + z + z' + b, N + x + \, N + b + x+\, -N - a, N + z + a + b + x+±, z-z'~ 
-z' + x + |, z + b, z, 2N + z + a + b + x + \, -a, N + z' + b + x + | 

-z + x + |, z' + b, z', 2N + z' + a + b + x + \ 
N + z' + b + x+\, 1-z + z', N + z' + a + b + x + \ 

+ a similar expression with z and z' interchanged . 



We now see that four gamma-factors cancel out, and also 



N + a + 1, -N -a 
a+1, —a 



(-!)"■ 



Further, we observe that the 4F3 factors are of the form 1 + O(^) as N — > 00. 
Indeed, this is true about any 



1-F3 



X, Y, Z,2N + T 
N + U.V, N + W 



with finite X, Y, Z, T, U, V,W; V 7^ 0, —1, —2, . . . , as follows from the series repre- 
sentation of 4F3. 

Taking this into account we obtain 



Q N {x 2 ) = {-1) N T(N + z + z' + b)T(N + x+ ±)T{N + b + x + ±) 



xT 



2N + z' + a + b + x + |, z - z' 
-z' + x + \, z + b, z, N + z' + a + b + x+\, N + z' + b + x + \ 

+ a similar expression with z and z' interchanged 
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Similarly, 

Qn-i(x 2 ) = {-lf-^N + z + z' + b + 1)T(N + x + |)r(JV + b + x + \) 

2N + z' + a + b + x+\, z- z 1 
-z' + x + \, z + b + 1, z + 1, N + z' + a + b + x + §, N + z' + b + x + 



x r 



x (1 + O(jj)) + a similar expression with z and z' interchanged j . 

Using a number of times the asymptotic relation T(M + c)/T(M) = M c (l + 0(-p-)), 
M — > +oo, c is fixed, we simplify the above expressions to get 

(-l) N Q N (x 2 ) 



T(2N + z + a + b + x+ i)r(2iV + z' + a + b + x+\) 



T(N + z + z' + b) 



x r 



z — z 

—z' + x + i z + 6, z 



(27V)— TV" 



-2z' — a — 6 



+ a similar expression with z and z' interchanged j 



and 



= T(N + z + z' + b + 1) 



^r(2iV + z + a + 6 + a; + ±)r(27V + z' + a + & + x + ±) 

(2W)^ jv^'-a-i^j + O(i)) 



x r 



z — z 



-z' + x+\, z + b+1, z+1 

+ a similar expression with z and z' interchanged 



Now as we compute Qn{x 2 )Qn-i(v 2 ) — Qn-i{x 2 )Qn{v 2 ) normalized by the 
square root of the product of four gamma functions as above, we observe that the 
terms involving nonzero powers of (2N) cancel out leaving a remainder of the form 

T(N + z + z' + b)T(N + z + z' + b+ i)7V-2(^'+a+6+i) . ( 1 ) ; 

where we used the fact that N ±( - Z ~ z '^O(jj) = o(l). Hence, the whole expression 
equals 



_ N -2(z+z'+a+b+l)Y 

( z(z + b)- z'(z' + b) 



N + z + z' + b, N + z + z' + b+l, z-z',z'-z 
z + 1, z' + l, z + b+1, z' + b+l 



z'(z' + b)-z(z + b) 



TT+°( 1 ) ■ 



\T(-z' + x+ i)r(-z + y + i) T(-z + x + i)r(-z' + y+\) 
Simplifying, we obtain 

N -2(z+z'+a+b+l) r \ N + Z + Z 1 + b, N + Z + Z 1 + b+l] 

[ z + 1, z' + 1, z + b + 1, z' + b + 1 J [L + 
tt(z + z' + 6) / 1 1 



' sin(^(z - z')) I T(-z' + x+ |)r(-z + y+\) T(-z + x+ \)T{-z' + y + \) 
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To complete the computation of the asymptotics of the correlation kernel, it remains 
to take care of the factors 

fcjv-i \fw{x - e)w(y - s) 



k N H N (x 2 - y 1 ) 



We see that 

fcjV-l 



= r 



= r 



2N + 1 — ai — a 2 — a 3 — a^, N + 3 — ai — a 2 — a 3 — a 4 
kN ~ " [ iV + 2 - ai - a 2 - a 3 - a 4 , 2A^ + 3 - ai - a 2 - a 3 - a 4 

l-z-z'-b,l-z-z'-b-N~ 
—z — z' — b — N, 1 — z — z' — b 



-(z + z' + b){z + z' + b+ 1)JV(1 + O(i)) 



and (assuming x ^ y) 
1 



x 2 -y 2 (TV + x + e- i) 2 - (iV + y + e- i) 2 2iV(z - y + o(l)) 



Further, let us consider the factor y/w(x — e)w(y — e). Observe that out of the 
eight gamma-functions that enter the expression 



w(x — e) 



N + 2e + z - \ 



T(-N - a -b-x+ 5)r(-a - JV - a: + ^)T{N + x+ ±)T(N + b + x + ±) 

1 

X T(z - x + \)T(z' -x+ \)T(2N + z + a + b + x+ |)r(2JV + z + a + b + x+\) 

we have already used the last two to normalize Qn and Qn-i above. The remaining 
contribution of yjw(x — e)w(y — e) equals 

|sin(7r(a + &))sin(7ra)| N 2a+1 (l + Q(±j) 

y/T(z -x+ \)T(z> -x + \)T(z - y + \)T(z> - y + \) 

Finally, using the formula of Proposition 8.2, we obtain, using the periodicity of 
sine several times, 



Hn-i — ± 



sin(7rz) sin(7rz') sin(7r(a + 6)) sin(7ra) sin(7r(z + z' + a + b)) 



sin(7r(z + b)) sin(7r(V + b)) 



2tt 4 sin(7r(z + z' + b))(-z - z' - b - 1) 

-z, -z', N,N + a, —N - z - z' - a - b,-z - b,-z' - b 
-N-z-z'-b 



Simplifying and using the fact that Hn-i must be positive, we obtain 

N- a {l+0{±)). 



|sin(7r(a + 6))sin(7ra . 
Hn-i — ^-z 1 



2ir 2 



N,N + a 
1 + z, 1 + z', 1 + z + b, 1 + z' + b 
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Gathering all the pieces together, we obtain that the correlation kernel K(x, y 
z, z', a, b | N), up to the factor (1 + o(l)), is equal to 



jy-2(2+ 2 '+a+6+l)p 

n(z + z' + b) ( 



N + z + z' + b, N + z + z' + b+l 
z + 1, z' + l, z + b + 1, z' + 6 + 1 

1 1 



sin(7r(z - z')) \T(-z' + x + |)T(-z + y + |) T(-z + x + \)T{-z' + y + \) 
x(-l)(z + z' + b)(z + z' + b + l)N ' 
| sin(7r(a + b)) sin(7ra)| 



2N(x - y) 

jy2a+l 



n2 yjT(z -x + \)T{z> -x+ \)T(z -y+ \)T{z' - y + \) 

sin(7r(a + b)) sin(7ra)| 



2tt 2 



N,N + a 
1 + z, 1 + z', 1 + z + b, 1 + z' + b 



N~ 



which, thanks to the asymptotic relation 

r 



N + z + z' + b, N + z + z' + b+l 
N, N + a 



= N 2(z+z'+b)+l-a( 1 + Q(l^ 



is readily seen to be asymptotically equal to K_ s 



\x,y\ 



z, ~z 



□ 



Similarly to the discrete hypergeometric kernel and the 3^2 kernel of §7, the 
correlation kernel K(x, y \ z, z', a, b \ N) also has a second form K(x, y \ z, z', a, b \ 
N) related to representing signatures in terms of the Frobenius coordinates. Using 
[B04, Theorem 5.10], it is easy to show that Theorem 8.3 proved above also implies 
the convergence of the second form K (x, y \ z, z', a, b | N) to the second form of the 
gamma kernel K samma (x,y \ —z, —z'). 

One can also compute the L-kernel, L = K(l — K)^ 1 , and consider the tail 
scaling limit of the correlation kernels and the L-kernel, but we will postpone the 
discussion of these issues until a later publication. 
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